Hamms: a misbehaving HTTP server for testing your clients
github.com
github.com
Without proper error handling, a wide randomly seeded crawl would hit some form of malformed response or bizarre header within 15 minutes at the most. I eventually gave up on trying to parse all the myriad odd behaviours and now just dump them all onto a blacklist and move the crawler on.
I tried to add some things as well that make testing easier when I am writing a client. Testing retry logic is hard for example because the client is making the same request a bunch of times in a row so you have to either mock the client request or make the server return different things based on the same request. I added a server that errors twice and then returns a 200, to make that easier to test.
Any plans on that?