For anyone that will need to update their own crawlers to match Google’s new javascript crawling, we’ve opened up our Prerendering engine that uses Headless Chrome at https://prerender.com. You can capture HTML, Screenshots, PDFs, or even HAR files from any web page with just an http request to our service. So it’s super easy to add javascript crawling to any crawler with Prerender.com (and it’s open source https://github.com/prerender/prerender).
For our Prerender.io customers, this announcement just means that Google will stop crawling ?_escaped_fragment_= URLs so they won’t request prerendered pages anymore. Instead, Google will just execute the javascript directly and index the result.
We’ve always recommended that our customers use the escaped fragment protocol, so it will be a smooth transition as Google slowly stops crawling the ?_escaped_fragment_= URLs. No changes need to be made if you are currently using Prerender.io. Keep an eye on our twitter (@prerender) and we’ll give updates on Google’s transition.
The one thing to look for when Google starts executing your javascript is keep an eye on your Google Webmaster Tools for your number of pages crawled by Google. In the past, we’ve seen that Google crawls much slower when executing the javascript themselves. Hopefully javascript websites don’t take a hit in number of pages crawled daily since that can affect large sites having all of their pages up to date in Google's index.