A lot of it comes down to the way NASA is organized. Generally it's big budget missions with a single spacecraft as the culmination of the mission. Rather than ongoing development programs. Also, NASA has this tendency to want to skip technology generations even though the end result tends to take longer to deliver and cost more than it would have if they would have just explored the intermediate technology space to start with (see: JWST and VentureStar). While we're talking about JWST, that's another big part of the way things are at NASA these days, that program has eaten most of the unmanned NASA budget and put a significant crimp on space science missions. It'll probably be worth it, but the cost is pretty extreme at this point. Also on the topic of JWST, I remember back when they were planning to have a flight demonstrator (Nexus) in the form of a Hubble class Earth-orbiting telescope using some of the NGST/JWST technology. Such a mission probably would have brought the risk of the JWST program more under control but for whatever reason they ended up cancelling it.
TPF is another example of planning farther ahead than is fully justified. As the program advanced the design became less and less fixed rather than more and more refined. This is an indication that they were designing something that was beyond their expertise and needed more intermediate research to figure out, but with the budget crunch and the lack of vision nobody could come up with good enough ideas to do so. NASA set their sights on deep nulling interferometry and it sounded like a really cool technology that might be capable of doing what they needed, but more and more it seems like that may not have been the best technology choice, and it led to a far too expensive system to fund.
If instead of JWST and SIM and TPF NASA had instead hitched it's wagon to a more incremental development approach, of launching Hubble or Kepler class telescopes in the $1 billion budget range every 2 or 3 years or so and using that program as a test bed for new technologies and new scientific investigations I think we'd be in a much better state right now and probably a lot closer toward having the capability to directly image exoplanets routinely. But who knows if that would have been the case.
On the plus side, the impending drop in the costs of orbital launch due to SpaceX et al will probably force NASA into a model such as that, so maybe we'll see if it generates better results.