> Anyone can index the web

Most, if not all, websites will block your bot after a couple of request even if it respects robots.txt.

Also they seem to have different rules for GoogleBot and different for yours.