George Elokobi knocked into his own goal after Enda Stevens' dangerous cross late into the game.
The home side could not unfortunately find a leveller and bowed out of the competition at the second round stage.
Doncaster shaded the early possession and had two chances in the opening ten minutes which they should have done better with.
Kyle Bennett and former loanee Reece Wabara shot from the edge of the box but goalkeeper Paddy Kenny saved both comfortably.
Efforts in the game were few and far between but Danny Philliskirk almost capitalised on a Doncaster error in defence. Enda Stevens attempted a pass back to Sam Johnstone and Philliskirk intercepted but was unable to get a shot away and the chance was lost.
With 20 minutes on the clock, Bolton loanee Kenny pulled off a wonderful stop to deny Theo Robinson from opening the scoring. The goalkeeper was straight out to smother the ball as the striker reached for the ball.
The home fans felt aggrieved just before the half hour mark as Jabo Ibehre looked to be pulled back in the area but the referee waved away the penalty protests.
The visitors went closest to opening the scoring on 32 minutes. Curtis Main prodded the ball into the path of former Latics midfielder Ritchie Wellens but the midfielder could not find a way past Kenny from six yards out.
The game livened up in the final five minutes of the half as both sides started to have clear cut chances on goal.
Athletic's best two chances of the game came in the last five minutes of the game. Jonathan Forte was firstly denied as Johnstone saved his volley and the ball broke for Ibehre but the striker could not direct his shot on goal from six yards out.
Latics shaded the early second half possession and were unfortunate not to take the lead on 53 minutes.
George Elokobi, who was making his first appearance since the end of September, rose highest from a Daniel Johnson free kick but he headed over the crossbar.
The home side were on top and started to dictate play. Liam Kelly, who had been instrumental in the middle of the park, blazed a volley over the bar from the edge of the box as Latics knocked on the door of their opponents.
Minutes later, Brian Wilson picked out Danny Philliskirk in the area but his header was well saved by Johnstone.
The visitors had a rare sight of goal just after the hour mark but Bennett's free kick was comfortably saved by Kenny.
Latics continued their domination of the second half but Philliskirk's drive from 25 yards was palmed away to safety by Johnstone.
Manager Lee Johnson threw striker Dominic Poleon into the action and he made an immediate impact, feeding Forte on the left hand side but the forward's shot struck the side netting.
Heading into the final ten minutes of the game, the visitors started to up the tempo and managed to get the winner with only three minutes of normal time remaining.
Enda Stevens played a dangerous ball into the box and a mix up between goalkeeper and defence saw Elokobi credited with the own goal.
With time against them, Latics went in search of a leveller and almost were rewarded a minute after the goal. Forte struck from the edge of the area and after three ricochets, the goalkeeper saved.
The fourth official announced five minutes of additional time with both sides having chances.
Rovers really should have put the game beyond doubt but Robinson missed the target from ten yards and Kelly's 35 yard volley struck the roof go the goal.
Time eventually ran out for the home side and Latics bowed out of the FA Cup at the second round stage.
Oldham Athletic: Kenny, B.Wilson, Kusunga, Elokobi, Mills, Kelly, Winchester (Poleon 68'), Johnson (Morgan-Smith 88'), Forte, Philliskirk, Ibehre.
Subs not used: Rachubka, Dieng, Mellor, Tidser, Dayton.
Doncaster Rovers: Johnstone, Robinson, Main, McCullough, Butler, Stevens, Keegan, Wellens, Wabara, Bennett (Forrester 75'), Coppinger (Furman 67').
Subs not used: Marosi, McCombe, Wakefield, De Val, McKay.
Goals: Oldham Athletic 0-1 Doncaster Rovers (Elokobi OG 87')
Attendance: 3,242 (452 visiting supporters)