Absolutely, but it turns out to be a really common misconfiguration found in default configs and tutorials. Definitely something to be aware of when configuring FastCGI.
5,156 karma · joined June 8, 2012
For this particular project I needed access to more than just the page itself - I needed to interact with scripts and have access to the page resources on a level which matched that of a browser. What you're seeing is only the tip of a much larger iceberg, for which efficiency wasn't the main objective.
You're correct that you can write more efficient spiders if you don't want to do anything fancy (indeed, I've written simple scrapers and spiders in various languages), but this project leant itself better to a full browser environment.
As a personal bonus for me, the author makes reference to the book "Gödel, Escher, Bach", of which I was somehow not previously aware. It sounds like it's right up my street.