> One suggestion I have is to be able to set cookies (much like you can set the user agent).
Thanks for taking a look and for the suggestion! You mean you'd be happy with an interface where you paste in the cookie value you wanted to use? Would the cookie ever have to be updated to keep working?
I'll need to think about the best way to implement this. For example, I could allow setting custom headers to support this and other customisations, or perhaps allow you to login via an iframe first to get the cookie data.
> Also, I'm curious as to why you decided to make a browser extension rather than a standalone web service. Any particular reason why?
A similar hosted web service would either be expensive or come with limitations to how many sites you can crawl and how often. For example, you might be limited to crawling a single production website and can only crawl it once a week.
With the extension approach, you can crawl unlimited websites, recrawl as often as you want, recrawl immediately whenever you want and easily crawl localhost + private sites. You don't need to wait a week to discover an issue has hit production and wait again to find out your attempted fixes didn't work. This way you can test for issues at all stages of development (localhost/development, staging and production) and get immediate feedback if your fixes worked. That's the workflow that I want to support.