Comment by 1vuio0pswjnm7 | Hacker News Reader

Comment by 1vuio0pswjnm7 | Hacker News Reader

"Which poses an interesting question: how do you create an API that should only be consumed asynchronously from a web page and never programmatically via a script?"

Web developers use JavaScripts to make HTTP requests to API endpoints. The data is being consumed by a script, programmatically. Unless Javascripts are neither scripts nor programs. Good luck with that argument.

There is a W3C TAG Ethical Web Principle that states web users can conusme and display data from the web any way they like. They do not need to use, for example, a particular software or a particular web page design.

2.12 People should be able to render web content as they want

People must be able to change web pages according to their needs. For example, people should be able to install style sheets, assistive browser extensions, and blockers of unwanted content or scripts or auto-played videos. We will build features and write specifications that respect peoples' agency, and will create user agents to represent those preferences on the web user's behalf.

With respect to the JavaScripts authored by web developers and inlined/sourced in web pages, web users have no control over them short of blocking them outright. As such, arguably they are not ideal for making HTTP requests to API endpoints. Unfortunately these scripts can be, and are, used to deny web users' agency.

Who cares?

He wants to provide a free service, which he is not obligated to do and costs money for him to do, for individuals to check their email. He never intended for bots to check a billion emails, and obviously doesn't want to pay for that.

That should be respected, and people failing to respect it is why we see the destruction of the open web with things like remote attestation as the only way forward.

Complaints like "but the open web" or "but my exotic browser" are honestly worth nothing against a potential solution for a real, pressing issue like spam requests and bots - if we want to keep the nice things we better start coming up with alternative solutions to the real problem, because corporations will decide the future for us if we don't. Ignoring it will not work out for us.

It often costs far more then money. In the this case bots were gaining data to better target future attacks. In the common spam case it costs real user's attention.

There definitely is an alternative solution here that preserves people's freedom to use whatever browser they want, but I'm not sure if anyone would like it.

Web services are intended for humans to use them, and all abuse also ultimately comes from humans directing computers to be abusive. Thus, rather than attaching a computer's temporary identity (an IP address) to the request, we should be attaching a human identity to it. Note here, that I don't care whether the request comes from John Smith in New York. I care about being able to ban you from the service if you are abusive, no matter how many computers you have at your disposal now or in the future.

There's lots of downsides I haven't got solutions for, such as, how do we stop sites cross referencing to reliably discover all users real identities (Google analytics would love this!), and so much more. But we'll have to give up something - if we do nothing we give up everything, maybe giving up only absolute anonymity could be preferred.

IP reputation doesn't seem to be much of a thing right now, but it might be the only way forward for a spyware-free web.

You can have a group of people sharing a block and collectively responding to abuse reports to slightly improve privacy. That at least shields you somewhat from the big tech firms.

Essentially a large number of virtual private micro-ISPs.

> IP reputation doesn't seem to be much of a thing right now

disclaimer: acquaintance works in the spam business. someone i tried to steer clear of, but was fascinated of what they told me.

IP reputation is a huge thing in the spam world. they pay top dollar for residential US/UK/etc IPs which they can then use for spamming others. and we're not talking one or two IPs. we're talking 100s of thousands of IPs being transacted daily globally. all for spam.

for anyone interested in seeing how low we've come, they have a huge convention in Las Vegas. one should visit it to better understand a field that has been growing like crazy but so few know it. apparently everyone is preparing for a big boom next year, with 10s of $B ready to be deployed.

IP reputation is absolutely a thing, as I understand it it's a large factor in Cloudflare deciding to put you in an infinite loop (or trigger the protection at all).

Yep. Even within a single VPN service it's often noticeable how many more CAPTCHAs (or outright blocks) you get via different servers

OK, but remote attestation isn't really a thing on the web (right now). And you can script a browser UI, anyway.

The point is to not get hung up on the client as a security boundary (it isn't, can't be, won't be), but to focus on the actual harm -- excessive use of the limited resources provided.

And you have to frame your security posture as rooted in the server side throttling, heuristics, et cetera. Flipping out because the client isn't what you expected isn't going to help (determined attackers can look like "vanilla" clients), and is just going to harm the long tail of actual users who are not bots.

It will also harm the short tail of bots who aren’t run by dedicated attackers.

> OK, but remote attestation isn't really a thing on the web (right now)

I mean, it's built into Safari and Cloudflare supposedly uses it for its CAPTCHAs: https://developer.apple.com/news/?id=huqjyh7k

It's not as invasive as Google's attempt to circumvent ad blockers, but it's still a remote attestation system implemented in the wild already.

> built into Safari

Is it really built into Safari, or are you referring to the iCloud option that "privately" helps reduce Cloudflare demands?

Private Access Tokens are built into Safari. Apple gives their devices a certain amount of tokens, and Cloudflare validates them. If Apple doesn't like your device, you won't get any more tokens. Cloudflare also hands out tokens if you install their browser addon.

The two companies are working together to make this an official web standard, but I haven't heard about it for a while. Maybe they're just laying low after seeing the blowback on Google's (worse) attempts at attesting devices.

I agree. Our public APIs are also massively queried. The number of queries is out of proportion to the legitimate traffic. Rate-Limiting does not work, because of the volume of different ips they send against you in parallel. Our servers are not designed for such peaks. What other choice do we have but to block them.

Did you consider API keys? Via email? That’s at least a natural throttle.

It's a public price API. There is no authentication required to list prices.

Host a static json file?

Preferably on a provider with no egress fees.

Could you explain why that API is meant to be public and what benefits it brings to human users? What kind of data does it return?

In addition to blocking/throttling, I have my services provide bad data to the abusing clients.

As in "valid" but false data? Please don't. If you really don't want to indicate rate limiting explicitly, then perhaps return an invalid body, or reset the connection or similar. False positives detecting humans as bots are very common, and even rate limits are often set well within human interaction limits. E.g. more than once I've triggered 429s by opening several e-commerce product pages in new tabs for me to ctrl+tab through and filter down. I also tripped a LinkedIn anti-automation system since I was looking through quite a lot of profiles on my first day to add people - luckily they handled this well, with a clear message explaining what was going on and support reaching out to me proactively (and lifting the restriction after a few hours)

If I block and throttle you already then yes, I reserve the right to break your downstream service by feeding you bad data.

> if we want to keep the nice things we better start coming up with alternative solutions to the real problem

Indeed. To state the obvious: the real problem are bad actors - no matter if they are nation states, cybercriminals or people running compromised devices - and their accomplices such as ISPs not responding to abuse reports.

As long as we don't get that under control (say, by threatening to cut offender countries and ISPs from the Internet and SS7 phone networks) we'll have to continue whack-a-mole'ing.

I still don't get why not just implement some basic usage rate limiting. Go two fold limit the rate to 2x or 3x the fastest they can manually use the service and limit users to only being able to burst like that for a bit.

You're going to have zero horid scraping once that's in place.

Because it doesn't work. Malicious users can circumvent the rate limiting by using botnets. If requests were somehow tied to the identity of the human operator rather than the particular computer they used, then yes, rate limiting would be all we need.

Offer a "free tier" with a global limit of N requests / second, and allow people to get an API key through email (for free also) which bypasses this, then rate-limit each API key.

TFA mentions that people were signing up with generated email addresses.

That doesn't solve the problem with bots in games. They are not hammering the API, they are automating things that shouldn't be automated and drives away non-bot players.

Dedicated scrapers have been using IP pools for more than a decade. Search for (residential) proxy pools and you’ll find a million vendors selling access. IP-based rate limiting stops cheap skids and screws over people behind CGNAT, that’s all.

> He wants to provide a free service, which he is not obligated to do and costs money for him to do, for individuals to check their email.

Not a good choice, why not make this a hashed db that could be distributed freely? Why are bots meant to be excluded? This is bad UX, I'd want to check my emails periodically in the background, but this "anti-bot" measure is meant to make me unable to do so and then demand payment. So this is openly a fight.

> are honestly worth nothing against a potential solution for a real, pressing issue like spam requests and bots

That is not an issue for me at all. Ok, let's face it - maybe I'm just on the other side. For me scrapers are very useful because they reduce the price needed to access the data as they introduce competition. For example price comparison sites are very useful.

Surely smart bots would call the API (https://haveibeenpwned.com/API/v2) rather than try to scrape the web form. HTTP calls to a web endpoint are much easier to scrape than whatever the website frontend decides to call. All you need to do is parse the Retry-After headed for the 429 error code and you can pretty much query away without worrying about CAPTCHAs.

If the service even provides such header or 429 status code at that. They could provide the 418 status code for fun. In this case – without checking the form implementation – I can assume that it provides a JSON response rather than a text response meant to be thrown into the DOM. Grabbing data from JSON is naturally easier but you could use a DOMParser for text content itself if it's sufficiently consistent.

The other thing about "waiting" is that bots may not want to do that, or maybe some sort of deadline is sooner than such waiting would allow.

To me, requiring a unique key to be input with the search, that is created after X amount of time (both provided by the initial response and increasing exponentially for subsequent requests) seems like it could be sufficient. If the next request is sooner than X then block them for Y amount of time for attempting to bypass allowable behavior. Allow like 5-10 emails then implement the wait-based functionality so that most non-bots would be fine. After all we're talking about blocking an endpoint only ever meant to be used via the website by actual users, typically they are not trying to check thousands of emails super fast.

I asked an online shop I frequent if they have an API, as they discount some inventory sometimes, to which they responded there is no public API. It's easy enough to find (using Inspect->Network->XHR) how they serve data, and with just five minutes of work, their raw JSON can be loaded in Python and polled for specific brands/categories/individual products, including anything flagged for a discount.

Programmatic? Yes. Against their wishes? Also, yes.

The guilt or bad feelings subsided after realizing I'll use more of their bandwidth how I was searching before via the browser. Using the API, I see when the last inventory update occurred, and only after this will I search their discount inventory.

(In any case, if they are on EC2, I cost them 3 to 8 cents per month, estimating high, which they have certainly recouped.)

Why have bad feelings at all. It’s a shop, and I’m assuming the end goal of you looking for discounts is to make purchases from that store?

More efficient.

They could (incorrectly) call you a "bot" but that's simply not true. You are human.

The amount of server resources you deplete is actually less than a modern graphical browser from an ad-sponsored team of software developers.

However perhaps you are not looking at some ads or submitting yourself to tracking or telemetry. Under the W3C web ethics principles you can consume the data from the web in any way you like.

It is your right to decide what HTTP requests you make. XHR/fetch in someone else's Javascript could be well-intentioned but too often it tries to take away some of that agency and transfer it to a web developer.

It's not so clear-cut. I have a Gitlab action that checks for the update every hour and then emails me when a match is found, so I technically have employed a bot. It saves me time and saves them bandwidth.

For tracking, they'll have to rely on old-fashioned "order history" and "inventory period". By these metrics, I would be baffled if they decided to block my access or close my account, even if I'm bypassing Google Tags/their AB test suite/social media analytics/ads for their other services.

Ignoring their nonconsent feels impolite, but they can dry their tears with my spent money.

What about botting in games? It's something that almost everyone is against so it's clear that the general opinion is not aligned with the W3C TAG Ethical Web Principle there.

I've seen many games die due to bots. If developers just straight-up allow them all the normal players quit.

> Javascripts

> 2.12 People should be able to render web content as they want

Yes, the choice between Chrome/FF/Safari/Lynx or some other remains

Any discussion not discussing how to deal with abusers is moot

Reply on news.ycombinator.com