Two late goals from Erhun Oztumer and Amadou Bakayoko won the game for the home side, despite Latics dominating large parts of the game.
Lee Croft was shown a red card in the first half for a late challenge whilst Brian Wilson received his marching orders in the second half for a second bookable offence.
Manager John Sheridan made two changes to the side which beat Northampton Town in midweek as Wilson returned to the side and Croft earned a start.
Wilson started the game in a midfield role and it would be Athletic who had the first chance of the game.
Ryan McLaughlin received the ball on the right hand side, whipped in a good cross into the box but Aaron Amadi-Hollaway could not connect properly with his cross.
Minutes later, Jason McCarthy fired over Connor Ripley’s crossbar from 25 yards as Latics restricted most of their efforts in the first half to outside the box.
Latics started the better of the two sides and on 13 minutes Aiden O’Neill curled towards goal but former loanee Neil Etheridge saved comfortably.
On 16 minutes came a controversial moment in the game as Croft received his marching orders from referee John Brooks.
The midfielder caught McCarthy late in front of the dug outs and the referee had no hesitation to head for his back pocket to send Croft off.
Latics needed to regroup. Wilson moved to full back as McLaughlin moved into midfield. Sheridan elected to stick with two strikers on the pitch.
Athletic almost took the lead on 20 minutes as Holloway headed into the path of Tope Obadeyi but the striker volleyed over the crossbar.
Both Kieron Morris and Isaiah Osbourne went close before the half hour mark before Latics went close again through an unlikely source.
The ball fell perfectly for Wilson 25 yards and he forced a good save from Etheridge as Latics continued to look dangerous in attack.
Two unbelievable saves followed from Ripley to keep the scores level at the Banks’s Stadium.
Firstly the goalkeeper denied Morris from point blank range, instinctively sticking out his hands to save from the midfielder, whilst five minutes later, he denied Osbourne from 18 yards as he got down quickly to save.
Athletic dominated a lot of the second half and created attack after attack despite their man disadvantage.
Latics won a free kick on the edge of the box on 55 minutes which Obadeyi took. He curled towards the far corner but it hit the wrong side of the post and went wide.
Five minutes later, Holloway’s fierce cross found Green in the centre but Walsall just managed to get back to deny the midfielder in front of goal.
Obadeyi was putting himself around in attack and was causing real problems for the Walsall defence. He evaded his man on 63 minutes and won a free kick for Latics.
That free kick was cleared by the home side and Ripley had to be on hand to save from Oztumer as they broke into the penalty area.
Obadeyi hit a free kick over the crossbar on 67 minutes before Peter Clarke rose highest at the back post but could not direct his header on target.
The home side started to turn the screw on the game as Latics tired towards the end of the game.
Kieron Morris fired over the crossbar as the visiting team enjoyed a sustained spell of possession in Oldham Athletic territory.
However, a long ball was cleared towards Holloway in attack and he appeared to have his shirt pulled which the referee did not see.
That decision would go against Latics as from the following Walsall attack, they won a penalty.
Clarke was adjudged to have fouled Simeon Jackson in the penalty area and Oztumer stepped up to send Ripley the wrong way and put Walsall ahead.
That goal came completely against the run of play but minutes later it would get worse for Latics as they were reduced to nine men.
Wilson and substitute Amadou Bakayoko came together and the referee awarded both a yellow card, Wilson’s second, which saw him dismissed.
However, Athletic still continued to come forward as Holloway fired wide on the volley and a goalmouth scramble resulted in a Walsall free kick.
Whilst Athletic piled men forward in the final minutes, they were caught on the break as Bakayoko smashed into the top corner from the edge of the box to seal the win for Walsall.
Walsall: Etheridge, Edwards (Bakayoko 69’), Laird, O’Connor, McCarthy, Chambers, Jackson (Makris 83’), Oztumer, Morris, Osbourne, O’Connell.
Subs not used: MacGillivray, Dobson, Randell, Roberts, Moussa.
Oldham Athletic: Ripley, Wilson, Gerrard, Clarke, Hunt, Croft, Green (Fane 64’), O’Neill, McLaughlin, Obadeyi (Erwin 86’, Holloway.
Subs not used: Kettings, Law, Flynn, Taylor, Ngoo.
Goals: Walsall (Oztumer 79’, Bakayoko 90+3’) 2-0 Oldham Athletic
Attendance: 6,003 (616 visiting supporters)