Using PhantomJS at scale

This is a particularly big issue for us because we spend more than 50% of our resources on serving robots (not humans!).

I wrote about this back in July of 2010, and the problem has only gotten worse: http://don.blogs.smugmug.com/2010/07/15/great-idea-google-sh...

It's a crazy problem to have, but we haven't found a solution. We can't just disappear from all the various indexes and collections that these robots represent.

I face a similar issue going forward. I think http://prerender.io has been posted to HN before - seems to be a pretty reasonable plug and play solution.