The 39-year-old, who saw off Lauren Davis 7-5, 6-2 in the first round, knocked out the fifth seed 6-3, 3-6, 7-6(4) to book her place in the third and avenge her loss to the Dutchwoman at Wimbledon last year.
After losing the first set 6-3, Bertens struck back to claim the second by the same scoreline.
Just as it looked as though she had averted an upset, though, the elder Williams sister raced into a 4-1 lead in the third set. She was soon pegged back while serving for the match but held her nerve in the eventual tiebreak.
'I made it!' - Muguruza becomes first Spaniard to win WTA Finals title
Elsehwere, Angelique Kerber was defeated 7-6(7), 6-2 by Anett Kontaveit.
The 23-year-old Kontaveit came from 5-3 down and saved a set point at 6-5 on the way to taking the opening set against the former world number one.
She broke Kerber twice to win the second set, winning the first nine points, and two holds to love gave her a 4-0 lead.
Swiss 12th seed Belinda Bencic retired with a foot injury after trailing former world number one Victoria Azarenka of Belarus 6-4 1-0.
Russia's Ekaterina Alexandrova, Zheng Saisai of China and Croatia's Donna Vekic were among other early winners.
Alexandrova turned back Carla Suarez Navarro of Spain 6-4 6-1, Zheng beat Viktoria Kuzmova of Slovakia 6-3 6-1 and Vekic, who will face Azarenka in the second round, was a 6-4 6-3 winner over Czech Katerina Siniakova.
Serena Williams is scheduled to play a night match in her first tournament since withdrawing from the Rogers Cup final in Toronto on Sunday due to back spasms.
The 10th-seeded American will face Kazakh Zarina Diyas.
'I can't believe it' - Kontaveit edges out Sakkari to make WTA Finals showpiece
Sakkari extends incredible top-five run - but who will win WTA Finals?