> Also, Google has increasingly made it difficult to automate searches

That's an understatement. Doing something as simple as

    inurl:"humans.txt"

...and consuming the first three pages showed me CAPTCHA. It was done via a browser, manually. Not all links were clicked.