The sport's governing body World Rugby and tournament officials said at a news conference on Thursday the decision had been taken due to the disruption, particularly to transport systems, expected from the typhoon.
The fierce tropical storm, predicted to be one of the most violent to hit the region in recent years, is expected to strike the Tokyo area this weekend.
England and France have already booked a spot in the quarter-finals but the decision to cancel Italy's match eliminates them from contention. A win over the All Blacks in Toyota could have taken the Italians through. New Zealand now top the group, ahead of South Africa.
With hosts Japan and Scotland still locked in battle for a place in the knockout round any decision to cancel that game will have a huge impact on the final placings. That decision will be taken on the day of the game.
Tournament rules state that any pool game that cannot be played is not rescheduled and instead is recorded as a 0-0 draw, giving each team two points, with no bonus points available.
If Japan's match in Yokohama is cancelled the hosts would go into the quarter-finals for the first time.
World Rugby Chief Operating Officer and Tournament Director Alan Gilpin said: "This is a complex and dynamic situation which we have been monitoring extremely closely with the assistance of our weather information experts.
"While making every possible effort to put in place a contingency plan that would enable all of Saturday’s matches to be played, it would be grossly irresponsible to leave teams, fans, volunteers and other tournament personnel exposed during what is predicted to be a severe typhoon."
He added that safety had to be the determining factor in these decisions.
"We fully appreciate that England, France, New Zealand and Italy fans will be disappointed, but we trust they will appreciate that their safety must come first," he said.
"They will be entitled to a full refund on their match tickets."
Super typhoon Hagibis, which was swirling some 490 km (306 miles) south-southwest of Japan's Chichijima island as of 9:00 a.m. (0000 GMT), is heading north toward Japan's main island and could make landfall on Saturday, the Japan Meteorological Agency said.
"Regardless of whether this typhoon will make landfall or not, it will make a grave impact. You'd better not obsessed with where the eye of the storm will be," an agency official told Reuters.
He said people should guard themselves against all the effects of the typhoon, including torrential rain, wind storms, storm surges and high waves.
With the cancellation of Saturday's game, England are confirmed as winners of Pool C and will play the runners-up in Pool D, which is likely to be Australia. France are set to face probable Pool D winners Wales.
It is the first time a World Cup match has been cancelled since the tournament started in 1987. The 1995 semi-final between hosts South Africa and France was delayed by an hour due to a waterlogged pitch, but famously went ahead after a team of cleaning ladies took to it with brooms.
England coach Eddie Jones was due to hold a news conference to discuss the decision later on Thursday.
Japan, who four years ago became the first team to fail to reach the knockout round after winning three pool games, would go into the quarter-finals as top of Group A if their match is called off.
Ireland would need any sort of victory in their final game against Samoa on Saturday, or a bonus point in defeat, to join them, with Scotland then eliminated.
Tokyo 2020 Olympic organisers also announced on Thursday that the test event for BMX cycling, originally scheduled to be held in the Japanese capital over the weekend, has been brought forward to Friday because of the impending typhoon.
Formula One officials say they are also monitoring the situation ahead of Sunday's Japanese Grand Prix.
(Additional reporting by Kiyoshi Takenaka; Editing by Tom Hogue/Peter Rutherford)