In "By the Waters of Babylon", what is the huge temple with its tunnels and caves?

By the Waters of Babylon is a short story which deals with a post world war civilization.  The characters we meet are probably descendants of the survivors.  The "temples" and the "caves and tunnels" are skyscrapers and subways that have remained over the long years since the destruction of civilization.