You're absolutely right that academics are torn between two things that they want:
1. open access for their research (both for altruistic reasons, because they want to spread knowledge as widely as possible, and for selfish reasons, because greater access to their research means more citations of their research, which helps them with career advancement)
2. publishing in the most prestigious journals they can, because that helps with career advancement.
The NIH, which funds all research into biology and medicine in the US, just cares about 1. above. They just care about maximizing distribution for taxpayer-funded research, as long as the distribution is done so in an economically sustainable way.
The NIH drove the open access mandate in 2008. My sense is that in the short term, driven mainly by the demands of funding agencies, all journals will move to 'open access models', where publishers pay to publish their research in the journals, and for readers, it's free to access. This satisfies both 1. and 2. above: academics get both open access and the validation of being published in prestigious journals.
Longer term, I think alternative credit metrics will replace the credit metric that's governed research for hundreds of years, i.e. the brand value of the journal you were published in.
I think when that happens, the power of journals will disappear altogether. Right now, the journals' role in driving the discovery of research has been disrupted a lot. In the pre-web days, people used to walk down to their libraries, and check out what had been published in the journals recently. In those days, the journals used to drive a lot of research discovery.
Nowadays, pretty much all research discovery happens online, and it's driven by things like search engines (Google, Google Scholar, Pubmed), and social platforms (Twitter, arXiv, Academia.edu; general communication technologies like email and IM are also used a lot).
The journals still have a strangle-hold on the validation of research. Which journal you were published in is still incredibly important, as far as the validation of your paper is concerned.
Other credit metrics are emerging, however. Citation counts are one metric, and that has a big role in driving research discovery on search engines like Google Scholar. Hiring committees are starting to look at the kinds of credit metrics that apply to general web content, e.g. page views.
As alternative credit metrics emerge, there will come a point when they do a good enough job at the validation of research that academics feel they no longer need to submit their papers to journals.
I wrote up some further thoughts on this here http://techcrunch.com/2012/02/05/the-future-of-peer-review/