Some of this is a little dated - for example there's not much point in managing a crawler yourself when you can just pay alexa to do it for you.
I still found a few useful tips though.
I still found a few useful tips though.
"Crawler. If you don't use an open source crawler, my advice is a super-simple multistep crawler. This is very important advice that will cut months off your development time, so if you ignore everything else, don't ignore this."