Lessons learned building a web application crawler

One of our engineers built a web-app crawler from scratch (why? enable users to quickly build a QA test plan). Jiri researched existing crawlers first and found nothing: he wrote about why build from scratch, plus technical challenges he ran into.