Maguire needed a big performance having dropped just out of the world's top 16, but it looked like his hopes would be dashed as Selby took a 2-1 lead thanks to breaks of 90 and 87 in the first and third frames.
Maguire kept in, however, responding with three frame-winning breaks of his own as the match progressed to 3-3.
And thereafter it was the Scot who held firm against the Jester from Leicester, making the most of a fluked snooker as he clinched the seventh and eighth frames of the match to earn a quarter-final spot against Graeme Dott.
Top 10 shots of 2021/22: No. 2 – O'Sullivan sinks outrageous red during Crucible epic
Stephen Maguire: I'll have to play better than that to win the tournament
Judd Trump, playing in the evening session, put in a majestic display to beat China's Zhang Anda 5-1 and make the quarter-finals.
Trump closed out the match with three consecutive centuries - including a 120 which was composed of 15 reds and blacks, the Bristolian failing to record a 147 only as he left himself partially snookered on the yellow and failed to pull off the pot with a serving effort.
"I was confident of making the 147 because I made one on the same table last year," said Trump.
"I just potted the (15th) black slightly too thin and the cue ball ran too far.
"If I can go out and play like that tomorrow I’ve got no worries.”
Martin Gould also enjoyed a huge win, beating Ben Woollaston 5-0, while Graeme Dott beat Barry Hawkins 5-2.
In the pick of the afternoon matches, World champion Stuart Bingham dug deep to come through a tense match 5-2 in the first round of the German Masters in Berlin against Ireland's Fergal O'Brien.
Bingham looked in fine form as he opened up the match with a break of 109 to get ahead, and after claiming a scrappy second frame still looked in control.
O'Brien hit back with breaks of 82 and 113 to level the match, however, and thereafter the encounter turned into a tense battle.
Bingham won the next to eke out a 3-2 lead, then pinched the error-strewn next two frames on the black to progress to the second round.
"I don't know how I've won it to be honest," Bingham told Eurosport after the match.
Stuart Bingham: I felt spaced out - I can't believe I won!
[WATCH: Snooker LIVE on Eurosport Player]
Next up for the reigning world champion is Ryan Day, who beat Liang Wenbo 5-3 in another close--fought match, going ahead in the match for the first time thanks to a 74 in the seventh frame, then coming through a tense eighth frame to win.
There were two surprise results during Thursday's afternoon session: world number 44 Mark Joyce shocked world number 11 Mark Allen 5-2, while Norway's world number 46 Kurt Maflin took down world number 13 Marco Fu 5-1.
[WATCH: Snooker on Eurosport: TV schedule]
'It's brilliant news' - Scottish Open set to return to Edinburgh
Top 10 shots of 2021/22: No. 3 – Williams creates Crucible magic with thrilling yellow
Share this article