> There used to be a time sites got punished if they treated searchbots differently..

Google is itself deep in the darkpatternmaxxing game now as well.