Carl Winchester had given Latics the lead going into the half time break but Joel Grant equalised for Yeovil ten minutes after the restart.
Brian Wilson was sent off with ten minutes of the game remaining and two minutes into stoppage time, Hayer struck to win the game for the hosts.
The game got off to a slow start with both sides struggling to create any clear-cut opportunity.
Yeovil thought they had taken the lead nine minutes into the game but former Liverpool striker Adam Morgan was ruled offside as he tucked the ball past Jake Kean.
Latics then started to dominate possession, forcing a succession of corners and keeping the ball well.
As the wind started to play havoc in the game, the game turned as the league's bottom side applied pressure to Kean's goal.
Kieffer Moore, Chigozie Ugwa and Morgan were all causing problems for Athletic's defenders but they failed to trouble the goalkeeper.
The home side looked dangerous from corners with the towering figure of Stephen Arthurworrey in the box causing problems.
Yeovil had their best opportunity to take the lead just before the half hour mark but Kean made a superb double save.
First, Arthurworrey's header was well saved and the goalkeeper picked himself back up to save from Ugwa.
Latics were having to withstand heavy pressure but Connor Brown eased that to the frustration of the home supporters.
With five minutes of the first half remaining, Latics had their best chance to take the lead through Winchester.
The Northern-Irishman raced through the middle but struck wide from just outside the box.
He did not rue that chance and gave Latics the lead minutes before the half time break.
Jonathan Forte held the ball up well on the left before crossing to Mike Jones. He guided the ball into the path of Winchester who finished confidently into the bottom to give Latics the lead at the interval.
Athletic started the second half confidently and had the added bonus of the wind which had effected them in the first half.
Winchester set off on a mazy run five minutes into the half which put pressure on the home defence but they held him up well.
Yeovil did find a way back into the game on 55 minutes as they equalised.
Morgan received the ball on the left, fired towards the top corner. Kean pushed the ball onto the post but the ball fell into the path of Grant who stabbed home the rebound.
Latics needed a response and brought on Conor Wilkinson in a bid to get three points from the game.
The Bolton loanee made a nuisance of himself in attack but was halted by the stubborn home defence.
Yeovil started to find control in the game and Kean had to be alert from the home attack.
With only ten minutes of the game remaining, Brian Wilson received his marching orders from the referee.
The full back picked up his second booking of the game after hauling down a Yeovil player in the centre circle.
That gave the home side the impetus to go forward and pepper Kean's goal.
Brown saved an almost certain goal on 84 minutes as substitute Hayter struck goalwards but he flew in front of the ball to clear.
Although down to ten men, Latics still continued to push for a winning goal Wilkinson and Rhys Murphy leading the attack.
Two minutes into additional time, Yeovil found the winner through Hayter.
With the ball bobbling around the area, the striker appeared in the box to rifle the ball past Kean.
Latics continued to press after the goal and thought they had levelled in the seventh minute of additional time but the referee disallowed the goal.
Yeovil Town: Loach, Smith, Foley, Moore (Hayter 69'), Grant (Twumasi 90+5'), Ugwa, Arthurworrey, Webster, Sheehan, Morgan (Leitch-Smith 87'), Shephard.
Subs not used: Krysiak, Berrett, Hoskins, Eastmond.
Oldham Athletic: Kean, B.Wilson, J.Wilson, Lockwood (Dieng 75'), Brown, Kelly, Winchester, Jones, Philliskirk (Murphy 69'), Poleon, Forte (Wilkinson 63').
Subs not used: Coleman, Kusunga, Truelove, Jacobs.
Goals: Yeovil Town (Grant 55', Hayter 90+2') 2-1 Oldham Athletic (Winchester 44')
Attendance: 3,917 (250 visiting supporters)