And this is only an issue for pages where you actually want to expose your content to search engines.
_escaped_fragment_ is a stupid hack that requires hardest bit of work needed for progressive enhancement, but provides none of the benefit.
Asking Timmy to "step up his game" is heartless, and it'd be unnecessary if we were better citizens.
I've never seen an example where Google crawled a page and included content that was loaded via ajax or an external javascript file. I have seen Google index content that is loaded from a script tag included in the html page.
However, from working with customers of BromBone, I know this. People have javascript powered webpages that were not getting indexed by Google. Google would crawl the page, and see no content. The pages were not in Google. After creating prerendered snapshots and serving them when the _escaped_fragment_ parameter was present, the pages showed up in Google.
And if you use html5 pushstate you could even serve the HTML captures to any bots. Eventually you also could easily add graceful degradation to your site serving snapshots for non js users. As of PoC for SEO4Ajax (http://www.seo4ajax.com), I built this application (http://www.appscharts.me) illustrating this purpose.
You are right about one thing, it is a pain to setup. Much harder than it sounds at first. After doing it a few times, I finally got sick of it and built http://www.BromBone.com. It's a service that takes generates the html snapshots for site owners and keeps them up to date. I hope it will let site owner keep all the positives and eliminate most of the new negatives.
I make this assumption because most of us don’t actually know how Google works, acting as a user’s proxy is in Google’s interest, and it is certainly within their technological capability. (Let me emphasize assume, again.)
This would require that the crawler actually renders sites, indexes the resulting DOM, and clicks what can be clicked. A headless Chrome, perhaps.
"Use a text browser such as Lynx to examine your site, because most search engine spiders see your site much as Lynx would. If fancy features such as JavaScript, cookies, session IDs, frames, DHTML, or Flash keep you from seeing all of your site in a text browser, then search engine spiders may have trouble crawling your site."
With regard to Google, however, they are explicit in their support for JavaScript rendered sites:
https://developers.google.com/webmasters/ajax-crawling/docs/...
Amongst many other links.