Getting to the moon from LEO takes less than half the delta-v that it takes to get to LEO, and WAY less fuel (because by this time you've dropped so much weight getting to LEO)
So the real problem will always be the stupidly inefficient method by which we conduct space flight. We send all of our supplies out of an enormous gravity well with a thick atmosphere.
Imagine what we could build if we already had everything in space for us already.
The only way we'll be able to become truly space-fairing is if we either become an astroid-mining civilization, or if we build something that lets us virtually ignore Earth's gravity well, like a space elevator.