Midfielder Kris Commons gave a much-changed home side the lead over the Ladbrokes Championship side in the 32nd minute with a stunning 20-yard drive.
The goal, however, was not the signal for the Hoops to kick-on and they toiled until midfielder Stefan Johansen grabbed an 87th minute clincher to put them into the quarter-final draw next Monday.
Boss Ronny Deila, as promised, utilised his squad and made six changes to the side which started in the 6-0 Ladbrokes Premiership win over Dundee on Sunday at Celtic Park.
Celtic fall to defeat at Real Betis, Rangers also beaten, Leverkusen win, Marseille held
Top scorer Leigh Griffiths was a notable absentee as Nadir Ciftci led the line in his first domestic start since signing from Dundee United in the summer with places for Tyler Blackett and Kieran Tierney in defence.
The top tier of Celtic Park was closed and a meagre crowd witnessed the home side survive an early scare.
In the 13th minute, moments after a great ball by Hoops wide-man Gary Mackay-Steven found no takers in the Rovers' box, Grant Anderson's header sent Craig Wighton through at the other end.
However, the Raith striker's powerful drive was parried by Parkhead keeper Craig Gordon with the danger cleared.
It was not the most pulsating cup-tie but just after the half-hour mark Commons interjected with a stinging left-footed shot from the edge of the box which sped past Kirkcaldy keeper Kevin Cuthbert.
The home side stepped up the pace momentarily and Raith defender Kyle Benedictus had to throw himself to block a goal-bound drive by Tierney for a corner, which came to nothing, as did another drive by Commons before the break which flew over.
The visitors might have levelled in the 58th minute when a long-throw by right-back Jason Thomson into the box fell to Ross Matthews, but his volley from 10 yards cleared the bar.
In the 65th minute, just after Gordon saved a decent long-range effort from Rovers midfielder James Craigen, Deila brought on James Forrest and Tom Rogic for Callum McGregor and Commons respectively and some chances arrived.
Mackay-Steven had a shot deflected past for a fruitless corner and then had an even better chance when Rogic set him up 10 yards out but Cuthbert made a fine save.
Forrest curled a shot just past the post from the edge of the penalty area moments later, as did Rogic with 10 minutes remaining.
The increasingly restless home fans held their breath in the 83rd minute when former Rangers striker Jon Daly, on as Rovers substitute for Iain Davidson, scooped a shot over the bar from six yards.
However, Celtic nerves were eased four minutes later when Johansen raced clear of the Rovers defence to knock the ball past Cuthbert, with time for Aiden Nesbitt to come on for Ciftci to make his debut.
Inside Football: Tchouameni, Nunez and more - Five possible under-the-radar PL deals
Celtic through to Europa League play-offs, St Johnstone exit to Galatasaray