One of the Olympic Games' fundamental values is supposed to be the promotion of health and harmony -- including ecological health and harmony with the land. The Lillehammer and Sydney games made big strides forward in terms of green building and sustainable energy use, but as Sustainable Style's magazine SASS reports efforts to green the Athens Olympics have fallen behind. We may have to wait for Vancouver to see what green Games could look like (PDF) -- the organizers there have promised LEED green buildings and "climate neutral" operations.
I have read some compelling arguments for permanently siting the summer games in Greece. This would end the expensive competitions between cities for the Games, even though that has brought about some nice showcase projects like Lillehammer.