Playing in her second tournament since the tennis season restarted, Williams won 7-6 (8-6) 3-6 7-6 (7-0) against the world No 72 in a gruelling contest that lasted nearly three hours.
Williams, who was 5-2 up in the final set before falling 6-5 behind, will face Maria Sakkari in the next round on Tuesday.
Naomi Osaka downs Elise Mertens to set up Victoria Azarenka final
An hour-long opening set was decided by a tie-break as both players held serve throughout. Williams took the breaker after saving a set point and then broke in the first game of the second set.
However, Rus won the next four games in a row to turn things around.
Williams looked out of sorts at the start of the third set but recovered from 0-40 down in the opening game to hold her serve.
That hold proved crucial as Williams broke in the next game to swing the momentum in her favour.
It looked as thought the match was soon heading for a conclusion as Williams opened up a 5-2 lead, only for Rus to break back and level at 5-5.
Rus then broke again and had a chance to serve for the match, but she couldn't close it out and Williams won every point of the decisive tie-breaker.
Johanna Konta breezed through her opening match 6-2 6-0 against Kirsten Flipkens.
The British No 1, who suffered a heart scare two weeks ago, didn’t face a single break point as she set up a last-32 meeting with Vera Zvonareva.
Stopping tennis for a day 'a step in the right direction' - Henin
Frustrated Serena compares loss to 'dating a guy that you know sucks'