Hacker News Reader

Comment by mahmud | Hacker News Reader

Google does execute javascript, and even some flash, but not on all its crawlers (which is very understandable; building a DOM for every page on the web is expensive, even for Google. I say "build a DOM" because abstract interpretation is still far off, specially for the web and its cruft of languages, standards and quirks.)

http://www.seomoz.org/ugc/new-reality-google-follows-links-i...

http://www.labnol.org/internet/search/googlebot-executes-jav...

View on HN

This is interesting, but it leads me to ask, what can you expect a JavaScript-enabled web crawler do? Load the page into the DOM, wait a few seconds for the dust to settle, start firing click events on everything and recording what requests are generated? It would seem a crawler could cause some really undesirable behavior on your site if that were the case. Imagine click events that post comments, delete things, downvote them, or generate emails. Since they are hidden behind JavaScript a web developer might expect a crawler not to try them...

There have been a few discussions about this topic in the past, and in the end it boils down to the developer being smart about the way he does requests.

I'm quite sure (wish I could remember where I read it) the crawler only actually performs GETs (which makes total sense because GET was always meant for retrieving and listing data), so a smart developer would use other HTTP methods (PUT, POST, DELETE) for actions that have modify existing application data.

Reply on news.ycombinator.com