But every passing day of the new coronavirus pandemic has brought more calls for a rethink of this year’s Games, scheduled to open on July 24
First, the cancellation in 1940. Then, the mass boycott in Moscow in 1980.
Forty years on, the upcoming Tokyo Games are the "cursed Olympics" once again, Japan's finance minister said, in remarks that could stir controversy at a time when his government is scrambling to quash speculation that the coronavirus epidemic could derail the world's biggest sporting event this year.
"It's a problem that's happened every 40 years - it's the cursed Olympics - and that's a fact," Taro Aso, who also serves as deputy prime minister, said in a parliamentary committee on Wednesday.
Japan had won the bid to host the Summer and Winter Olympics in 1940 - in Tokyo and Sapporo, respectively - but both Games were cancelled due to the Second World War.
A close ally of Prime Minister Shinzo Abe and a former premier himself, Aso is known for his staying power despite his penchant for gaffes that have insulted people, including doctors, women and Alzheimer's patients, over the decades.
Abe has staked his legacy as the longest-serving Japanese leader on staging a successful Games and bringing a massive jolt - estimated at $2.3 billion - to the stagnant economy with tourism and consumer spending.
But every passing day of the new coronavirus pandemic has brought more calls for a rethink of this year's Games, scheduled to open on July 24.
The International Olympic Committee, along with Tokyo's organising committee and Japanese government, has said it is not considering a cancellation or postponement, even as other major events have been postponed, including, on Tuesday, soccer's Euro 2020 and Copa America.
The new coronavirus has so far killed more than 8,200 people and infected more than 200,000, with the epicentre now in Europe. Japan saw 40 new infections on Wednesday for a total of 1,629 cases, including about 700 from a cruise ship quarantined near Tokyo last month, Kyodo News reported.