Selby was on the back foot throughout, but fought hard to get into position to win the ninth frame and make the score 5-4.
But a crucial missed cannon left him with a long red that he failed to make, and O'Sullivan produced a superb clearance to the black to win the match.
“I wasn’t feeling technically great out there but I still had to go for my shots and try to make things happen,” said O’Sullivan.
Northern Ireland Open
'I can't wait' - Higgins pumped to face Allen in Belfast final showpiece
“I wanted to keep attacking and keep my opponent on the back foot. My long game was really good and I stayed patient."
Selby, who beat O'Sullivan in the World Championship final in 2015, had admitted that he was particularly looking forward to one of the game's most notorious grudge matches, with little love being lost between the two over the years.
But O'Sullivan played down that aspect of their relationship after the match.
"Mark is the only player who can win tournaments when he’s not at his best because tactically he is unbelievable and he’s the hardest match player on the circuit. I watched all of his matches during the UK Championship and I realised how difficult he is to play for every opponent – he’s like granite," he said.
"I realised he doesn’t only play like that against me. I admire his tenacity and his will to win, I’ve got massive respect for him. We’ve had a bit of grudge and needle over the years and I have probably taken it personally, but not any more because I know he’s a great player.”
Watch this event on Eurosport Player
But it was O'Sullivan who took charge of the match from the start, winning three of the opening four frames.
Selby pulled one back after the interval only for O'Sullivan to make a century in the next to and follow up with another to lead 5-2.
Again, Selby closed the gap with a 68 break to trail 5-3 and seemed likely to push the match all the way.
But O'Sullivan seized on his opponent's mistake to close out the match, showing off some outstanding potting despite occasional shaky positional play - including on the final blue - as he made his way into the semi-finals.
Snooker LIVE on Eurosport: TV schedule
In the evening match, Barry Hawkins made the final four courtesy of a one-sided 6-2 win over Mark Allen.
“It’s a great feeling to be in the semis of such a big event,” said Hawkins, who had never won a match in five previous appearances at the Masters.
“I played solid. I kept him out and didn’t give him many chances. My safety was good while he played a few loose ones and left me chances."
Northern Ireland Open
Higgins v Allen - Northern Ireland Open final live updates
Northern Ireland Open
'Messi can maybe make the game look that easy' - O'Sullivan wowed by Higgins