The 34-year-old from Luxembourg, who beat Britain's Aljaz Bedene in the third round, fired down 30 aces as he sent the two-time champion packing in a brilliant display of power in just under five hours.
Muller won the first two sets 6-3 6-4 before the Spaniard fought back to take the next two sets 6-3 6-4 to set up a thrilling finale.
The 31-year-old Nadal saved four match points in a 135-minute minute fifth set in which he successfully served to stay alive nine times.
At the 10th time of asking, however, and with thousands of transfixed fans watching the drama unfold on the screen on the hill and on the edge of their seats on Court One, Nadal buckled under the relentless pressure exerted by an inspired Muller.
"I'm just glad it's over," the 34-year-old Muller, who claimed a second Wimbledon win over Nadal having beaten him at the All England Club in 2005, said.
"I did really well in the first two sets, then Rafa stepped it up. It was a big battle. When I had the last two match points, I thought I just had to give it a shot."
Muller will face Marin Cilic in the last eight, with No 1 seed Andy Murray, who saw off Benoit Paire on Monday, his potential semi-final opponent.