I figured old data was uninteresting, but TIL that crawling data yourself is not impossible. I mean you have to come from a thousand IPs, browser UAs and cookie states to find everything thanks to the sad state of things, but wow, I didn't realize 200Mbps could do so much. :D
As for the errors et al, that makes me think, I wonder what would happen if you saved that data and folded it into the seed list (with maybe 3 or 4 URLs up the chain leading to that link) for next run.
Agh... so you already have leeches :( all I can suggest is maybe mixing up your HTML, or maybe putting your site behind CloudFlare and, if there's an option(?), setting the attack detection sensitivity all the way up. I have no idea if they can do this.
10 queries/sec... that's... not going to be too great on the hardware :( I mean these people can just crawl the info themselves and get more up-to-date content >.< wow
By "free" API, I don't mean free-for-all "here, have at my bandwidth/hardware" xD - I mean more in the sense that curious shy CompSci types can maybe autogenerate themselves a basic-level API key to experiment with ultra-low-rate data requests here and there, and request a threshold upgrade on their key if they think they have an interesting/justifiable use for the data.
And it would be great if this could be a revenue stream! Have you ever featured the site on HN?
PS. Constructive criticism on design: I would personally alter the page styles a little; it currently looks nothing like what I associate "search engine homepage" with based on what I've learned to expect, and the page style, while very nice, makes me feel like the site is a blog, not a dynamic search engine.