So imagine what bots often don't have.
Adding JS interaction and cookies takes more effort on the part of the programmer writing a bot.
So yeah, you'd look a lot more like a robot. How else would you quickly differentiate between human vs non-human based on a single request, or even a collection of requests over time? It's a game of stats at scale.
from splinter import Browser b = Browser() b.visit('http://google.com') b.fill('q', 'browser automation') btn = b.find_by_name('btnG') btn.click()
Not exactly 'more effort'...
require "selenium-webdriver" driver = Selenium::WebDriver.for :firefox driver.navigate.to "http://google.com" element = driver.find_element(:name, 'q') element.send_keys "Hello WebDriver!" element.submit
https://code.google.com/p/selenium/wiki/RubyBindings
Writing a bot with js and cookies is trivial, but it definitely won't defeat these tools. They probably look for time between actions or track mouse movements, stuff that makes the bots super inefficient.
Right now google and bing will run sites with JS enabled to see the DOM after any JS changes take hold. Usually these crawls aren't nearly as often as the general crawling, because there is quite a lot more CPU/Memory overhead to such utilities. I can't speak for splinter, but similar tools in node or phantomjs have a lot over overhead to them.