Selby was on the back foot throughout, but fought hard to get into position to win the ninth frame and make the score 5-4.
But a crucial missed cannon left him with a long red that he failed to make, and O'Sullivan produced a superb clearance to the black to win the match.
“I wasn’t feeling technically great out there but I still had to go for my shots and try to make things happen,” said O’Sullivan.
German Masters
On the pioneering work of Ding Junhui and how one gamble transformed a sport
“I wanted to keep attacking and keep my opponent on the back foot. My long game was really good and I stayed patient."
Selby, who beat O'Sullivan in the World Championship final in 2015, had admitted that he was particularly looking forward to one of the game's most notorious grudge matches, with little love being lost between the two over the years.
But O'Sullivan played down that aspect of their relationship after the match.
"Mark is the only player who can win tournaments when he’s not at his best because tactically he is unbelievable and he’s the hardest match player on the circuit. I watched all of his matches during the UK Championship and I realised how difficult he is to play for every opponent – he’s like granite," he said.
"I realised he doesn’t only play like that against me. I admire his tenacity and his will to win, I’ve got massive respect for him. We’ve had a bit of grudge and needle over the years and I have probably taken it personally, but not any more because I know he’s a great player.”
Watch this event on Eurosport Player
But it was O'Sullivan who took charge of the match from the start, winning three of the opening four frames.
Selby pulled one back after the interval only for O'Sullivan to make a century in the next to and follow up with another to lead 5-2.
Again, Selby closed the gap with a 68 break to trail 5-3 and seemed likely to push the match all the way.
But O'Sullivan seized on his opponent's mistake to close out the match, showing off some outstanding potting despite occasional shaky positional play - including on the final blue - as he made his way into the semi-finals.
Snooker LIVE on Eurosport: TV schedule
In the evening match, Barry Hawkins made the final four courtesy of a one-sided 6-2 win over Mark Allen.
“It’s a great feeling to be in the semis of such a big event,” said Hawkins, who had never won a match in five previous appearances at the Masters.
“I played solid. I kept him out and didn’t give him many chances. My safety was good while he played a few loose ones and left me chances."
German Masters
Selby in danger of missing Players Championship as Vafaei secures spot in top-16 event
Shoot Out
‘Knock me over!’ – Vafaei produces ‘perfect frame’ to win Shoot Out