STS-135 Live: The Last Shuttle Mission

You mean $2.4 Trillion?

The stuff I have read points to the X-Prize as the catalyst, though you are likely right in that there was a combination of factors that allowed these enterprises off the ground.

Certainly a load of geeks flushed with cash didn't hurt at all.

If you weren't following the development of "new space" then SpaceX's achievements would seem to come out of nowhere for seemingly no reason, but that's not the case, there was a lot of buildup.

The main reason for "why now" is the end of the Cold War. During the Cold War there were geopolitical military implications to building launch vehicles, and there was a huge brain drain (or brain sink if you like) of aerospace talent into military projects.

In the 1990s there was a very substantial boom in "new space" companies (orbital launch startups). A lot of that growth was fueled by the growing venture capital from the dot-com boom, however in many ways this was a very flawed boom that did more to hurt new space than help it. For one, much of it was based on a then emerging launch market for low-orbit comsat constellations such as Iridium, Globalstar, and Teledesic (which, of course, were in turn vastly fueled by the tech boom).

More importantly though the heavy influence of VC firms in providing investment funding for many new space companies twisted the industry in weird ways. The way VC works in tech based startups is often to seek out highly disruptive companies that are both high risk but that have a high chance of an enormous return if they succeed. This doesn't work as well with companies founded on physical engineering and it didn't help much for launch vehicle development. A lot of the early new space companies were founded on highly innovative and unconventional launch vehicle designs (hybrid jet/rocket planes, rocket helicopters, RLV SSTOs, etc.) which were at best several generations ahead of the current state of the art (and at worst highly impractical or unworkable). When the dot-com bubble burst the comsat constellation companies did as well, bringing down many new space startups with them.

However, out of those ashes a new generation of launch startups grew. These new startups didn't have the luxury of fantastical potential revenue streams from a comsat boom, they had to build their businesses on hard-nosed accounting and either more proven or more likely revenue sources such as traditional launch business (still a very big market) or space tourism. This spurred companies to be more practical in their engineering choices, to use iterative development practices, and to spend more of their efforts in competing with traditional launch providers. In combination with the X-Prize (which had been created in 1996, in the midst of the first generation of the new space boom) this helped create the current crop of space startups we know and love today. Companies like SpaceX who built a very traditional expendable rocket but did so using a competitive spirit in terms of expense and complexity not seen with most of the launch vehicles built via multi-billion dollar cost-plus government contracts. Companies like XCOR and Virgin Galactic/Scaled who are paving the way in sub-orbital space tourism with very elegant and simplified reusable rocket ships designed for simplicity and robustness of operations. Companies like Blue Origin who took up an enormously promising reusable launch vehicle design that the government abandoned (Delta Clipper), or Bigelow who took up an enormously promising space station design also abandoned by the government (TransHab). Companies like Masten and Armadillo showing that you don't have to be a billion dollar aerospace megacorp to do good work or blaze new trails in rocketry.

Despite a late and rocky start, the startup/barnstormer phase of manned and unmanned rocketry has finally begun in earnest. This phase propelled the design of the airplane from experimental and impractical craft like the Wright flyer to advance within 2 decades to the industrial, commercial, and military workhorses made of metal, able to carry a dozen passengers, and fly across or between continents that were built by the dozens or hundreds in the 1910s and '20s. And then a decade after that to craft like the DC-3 and the Spitfire. Launch vehicle development may not advance at quite that pace but it is nevertheless an enormously exciting time filled with a great deal of promise for anyone interested in spaceflight.

The Shuttle is a fairly classic example of "2nd system syndrome". It was designed to replace every single launch vehicle currently in existence and to become the backbone of manned spaceflight, cargo launches, and military spaceflight. Large payloads were to be assembled in orbit in pieces (a la ISS). Comsats and suchlike were to be launched in the cargo bay far cheaper than any expendable rocket could manage. Military missions were to be conducted with very little prep-time from Vandenberg AFB with polar missions that would fly over Western Russia (Moscow, Leningrad, etc.) and then return in one orbit back to the launch site. Since the Earth would have rotated under the Shuttle's orbit this would require an immense cross-range flight capability (about a thousand miles), this requirement forced the orbiters to have exceptionally large wings and thus a stronger airframe to support them and a more extensive re-entry shielding, both of which significantly increased the dry weight of the vehicle as well as the total area of extremely vulnerable thermal protection systems (such as the much larger wing leading edge carbon-carbon shielding, which led to the disintegration of Columbia on re-entry when it was damaged by iced foam during takeoff). That cross-range flight capability was never used in practice as polar military missions from Vandenberg never came about, though the requirement imposed a heavy burden on the Shuttle's capabilities and costs.

The Shuttle was also designed to haul an airlock, rendezvous equipment, and a robotic arm, even though such capabilities are not necessary for every mission, though again the requirements affect the capabilities and cost of every mission. On paper each orbiter was to have flow dozens of times a year, this was a key element of lowering launch costs. In practice the orbiters were not truly reusable and were more refurbishable, the main engines were replaced on each flight, all of the consumables were flushed, the cargo bay liner was removed and replaced, the tile-based thermal shield had to be extensively inspected, etc. in a months long process that steeply limited the flight rate of the orbiters and vastly increased launch costs (ultimately making the Shuttle one of the most expensive launch vehicles of all time).

Overall the Shuttles tried to satisfy too many mission requirements and ultimately did not achieve any of them well. And in the process a hugely compromised system which was vastly more expensive and vastly less capable than previous systems was built.

I imagine it's also quite hot...