Brailsford has masterminded his team's domination of the race over the past decade since Wiggins became the first British man to win the Tour in 2012.
Chris Froome has won four times, Geraint Thomas once and last year Colombian Egan Bernal claimed the overall jersey in the team's first season as Ineos rather than Team Sky.
Froome and Thomas were both left off the Ineos Grenadiers team for this year's race and Bernal blew up on the Col du Grand Colombier on Sunday, losing eight minutes on the leaders to all but end his challenge for the yellow jersey.
"It's hard to stay on top every year and this year they've just not got it right," Wiggins told Eurosport.
"I don't know what's happened there. For a team that's performance orientated and such planning that goes into their seasons, it's just not worked for them for one reason or another. Had it been football, Dave would be out."
Wiggins said it would be dangerous to assume Brailsford's once all-conquering outfit were in terminal decline though.
"Dave has been here before and they'll restructure and re-plan and come back next year with a full line-up. With the world going back to normal, that will help," he said.
The Dutch-based Jumbo Visma team have dominated this year's race so far with Primoz Roglic the overall leader ahead of Tuesday's mountain stage. It ends in Paris on Sunday. (Reporting by Martyn Herman, editing by Pritha Sarkar)