Comment by zb3 | Hacker News Reader

Comment by zb3 | Hacker News Reader

Parent Full thread

> He wants to provide a free service, which he is not obligated to do and costs money for him to do, for individuals to check their email.

Not a good choice, why not make this a hashed db that could be distributed freely? Why are bots meant to be excluded? This is bad UX, I'd want to check my emails periodically in the background, but this "anti-bot" measure is meant to make me unable to do so and then demand payment. So this is openly a fight.

> are honestly worth nothing against a potential solution for a real, pressing issue like spam requests and bots

That is not an issue for me at all. Ok, let's face it - maybe I'm just on the other side. For me scrapers are very useful because they reduce the price needed to access the data as they introduce competition. For example price comparison sites are very useful.

Surely smart bots would call the API (https://haveibeenpwned.com/API/v2) rather than try to scrape the web form. HTTP calls to a web endpoint are much easier to scrape than whatever the website frontend decides to call. All you need to do is parse the Retry-After headed for the 429 error code and you can pretty much query away without worrying about CAPTCHAs.

If the service even provides such header or 429 status code at that. They could provide the 418 status code for fun. In this case – without checking the form implementation – I can assume that it provides a JSON response rather than a text response meant to be thrown into the DOM. Grabbing data from JSON is naturally easier but you could use a DOMParser for text content itself if it's sufficiently consistent.

The other thing about "waiting" is that bots may not want to do that, or maybe some sort of deadline is sooner than such waiting would allow.

To me, requiring a unique key to be input with the search, that is created after X amount of time (both provided by the initial response and increasing exponentially for subsequent requests) seems like it could be sufficient. If the next request is sooner than X then block them for Y amount of time for attempting to bypass allowable behavior. Allow like 5-10 emails then implement the wait-based functionality so that most non-bots would be fine. After all we're talking about blocking an endpoint only ever meant to be used via the website by actual users, typically they are not trying to check thousands of emails super fast.

Reply on news.ycombinator.com