Because there was no incentive. Space was something governments did and after the USA won the space race, interest waned. SpaceX proved there is a commercial market. Now we are witnessing the start of a commercial space boom. https://en.wikipedia.org/wiki/List_of_private_spaceflight_co... Look at the number of launchers in development. Competition = innovation. And once space mining has been proven as a concept, then the real boom will begin. I'm optimistic and expect all of this to happen in the next 20 years.
>What is this new propulsion technology that you anticipate?
Ion drives?, scramjets?, something that hasn't been discovered yet?, ... I honestly don't know but I doubt we will still be stuck with chemical rockets a 100 years from now.