Mike Jones gave Latics the lead in the first half with a superb strike into the top corner.
However, in an electric final 10 minutes of the game, Swindon Town equalised through Nicky Ajose’s cool chip over David Cornell.
Latics’ heads did not drop and two minutes later, Danny Philliskirk put them back ahead and won the game with a sumptuous finish as Dunn claimed his first three points as manager.
Dunn made two changes to his starting line up as Michael Higdon replaced Dominic Poleon and Brian Wilson took the place of the suspended Jonathan Burn as Latics lined up in a 4-4-1-1 formation.
It was the hosts that probably started the better of the two sides with Ben Gladwin going close with an early effort.
Bradley Barry was the next to go close, shooting wide from the edge of the box but Latics did start to come back into the game.
Swindon were playing a dangerous game at the back and Latics were putting them under increased pressure in good areas of the field.
Latics could have taken the lead on 10 minutes had it not been for Swindon goalkeeper Lawrence Vigouroux.
The stopper did well to deny the impressive Carl Winchester from six yards and got straight back up to save Michael Higdon’s header on the line.
Two minutes later, Jay Fulton delivered a teasing ball into the box which Philliskirk almost latched on to but it was just ahead of the forward.
Winchester was booked for a foul a minute later, whilst Higdon tried his luck from 35 yards but Vigouroux saved well.
Midway through the first half, Latics almost fell behind after David Cornell clashed into a defender and dropped the ball but B.Wilson was back to clear the danger.
Latics needed a lift after the game started to slow down and they took the lead on 34 minutes through Jones.
Winchester, who was off balance, excellently laid the ball into the path of Jones and he advanced onto the edge of the box and unleashed a curling effort into the top corner for his first goal of the season.
Athletic were in complete control after the goal and started to knock the ball around with purpose.
Five minutes before the break, a 15-pass move, moving the ball from back to front, ended in a shot from Danny Philliskirk, but it just went wide of the goalkeeper’s far post.
With the half time whistle approaching, Fulton received a harsh yellow card and the referee awarded Swindon a free kick in a promising position.
However, Yaser Kasim’s free kick did not trouble Cornell and Latics went into the half time break ahead.
It was much the same in the second half for Latics as they controlled much of the ball in the opposition half.
Latics were limited their opponents to shots from distance with Raphael Rossi-Branco trying his luck from 40 yards, which failed to trouble Cornell.
Dummigan was proving to be a handful on the right-hand side, as well as his fellow Burnley loanee Daniel Lafferty who was proving a similar threat on the left.
His cross on 55 minutes found Higdon in the penalty box and he nodded back to B.Wilson to shoot but it went wide of the post.
Athletic were frustrating their opponents and just before the hour mark, ironic cheers went around the ground as Swindon midfielder Drissa Traore shot but Cornell comfortably saved.
Minutes later, a succession of corners almost saw Latics extend their lead but Jones’ corner was saved by the goalkeeper.
Winchester and Philliskirk were causing the visitors a lot of problems. This time they combined to cross for Higdon but the Swindon stopper saved.
Latics were happy to let the home side have the ball in their own half and pressure when they entered Latics’ half and it proved to be a good game plan for Dunn’s side.
The home side started to dominate Latics going into the final ten minutes of the game as they attempted to get something out of the game.
It proved to be a nervy final ten minutes of the game.
Swindon did manage to pull a equalise on 85 minutes through Ajose. The striker saw his first shot saved by Cornell but he put away the rebound, chipping the goalkeeper.
However, two minutes later, Latics were back in front in style as Philliskirk notched his second goal in as many games.
The forward latched onto a through ball, took one touch and volleyed past the goalkeeper to win the game for Latics.
Swindon threw everything forward to get something out of the game but it was Latics that came away with the deserved victory.
Swindon Town: Vigouroux, Barry, Branco, El-Abd, Ormonde-Ottewill, Brophy (Balmy 62’), Traore, Rodgers, Kasim, Gladwin, Ajose.
Subs not used: Belford, Bangoura, Ojamaa, Stewart, Iandolo, Randall.
Oldham Athletic: Cornell, Dummigan, J.Wilson, B.Wilson (Dieng 78’), Lafferty, Kelly, Jones, Fulton (Yeates 90’), Winchester, Philliskirk, Higdon (Poleon 80’).
Subs not used: Coleman, Eckersley, Croft, Fuller.
Goals: Swindon Town (Ajose 85’) 1-2 Oldham Athletic (Jones 34’, Philliskirk 87’)