The idea stays: you can jerry-rig a web crawler from pedestrian, readily available parts, and quickly test the idea for which you wanted to crawl the Web.

The point is that bash, xargs, wget, make, etc are well-understood, well-maintained tools. They are not even necessarily simple, but they likely contain very few unknown unknowns.