Many comments so far seem to try to handwave away the 90 % failure as somehow "optimal" in the system, which seems absurd to me. It is clearly not advantageous for individual companies to keep a drug candidate alive long enough for it to fail in stage III or IV. One obvious question is why they don't and it is very, very tempting to speculate that it's because the problems are getting harder, we are targeting novel mechanisms etc. Again, I think this misses a simpler explanation:
As with many cases where companies make seemingly bad decisions, I think a lot of the explanation lies in system dynamics. Think about the incentive structure inside large pharma companies - it is generally not a career advancement move for a project manager to kill the drug candidate they oversee. It is career advancing to get it approved for the next stage. What could possibly go wrong in this world?