"Do you see all these things?" he asked. "Truly I tell you, not one stone here will be left on another; every one will be thrown down."
Matthew 24:2
After all the woes and condemnations that Jesus had declared, there really wasn't a lot for the Jew to hang his hat on so to speak. They had been stripped completely bare, and knew it wasn't just directed at the Pharisees; it was directed at all.
What do you do when everything you based your identity in was laid waste? Many search for vindication; they look for tangible signs of that which refers to anything that still maintains their stature as a people. In this situation, buildings seemed like a good idea to the disciples.
But Jesus even takes that away with one prophetic word. He then paints a picture that would put a scene from Mad Max to shame. No. Buildings, triumphs, wealth, culture - all are temples of vanity in the eyes of the King. It's only when the gospel of the Kingdom is preached to the whole world - only when people have chosen to focus on that one ambition will His coming and the end of the age be imminent. Only then will shrines and temples become irrelevant and the Son of Man returns to establish His manifest reign.