Hacker News Reader

It's not the "same" crawler. Probably each thread or each cluster machine instance of the crawler hitting it independently.

That's still the same crawler system though. And it's lazy engineering to not build in something to track when you last requested a url.

And it's quite a trivial feature at that.

I sincerely doubt that search engines run their crawlers on a single machine and they got it figured out.