Hacker News Reader

Comment by brnt | Hacker News Reader

Seems like they could also let one request per hour through, say, and then serve up the rest a 429.

Users trying to clone their project would hit an almost certainly up to date Google cache and thus be happy and sr.ht save on pretty much all that traffic and thus be also happy.

View on HN

Why the burden of making this google-specific rate-limiter should fall on Sourcehut ? They are a small team and a user-agent-per-repo specific limiter have rare uses.

Google should definitely fix their shit, but when we see that a googler said "it would be a fair bit of extra work for us to read robots.txt", I understand the frustration on Sourcehut side, and banning an user agent is usually just a single line of configuration to add.

I think the actions here are meant to generate attention. Google is acting in bad faith and needs to be called out.

I thought that too, but then sr.ht needs to keep state which IP made how many requests already, and as the post mentions google makes these requests from many different places. So they would have to count requests from the specific user agent and by that point it is again special purpose work and load on sr.ht (whereas general rate limiting per IP might be a good idea anyway).

Per blogpost:

> git clone requests with a GoModuleMirror User-Agent will receive a 429

So only one counter seems to be needed. Sure, it's doing work for an adversarial party, but it's not much extra work either.

Ah, not sure I understand then. Would it still need one counter, or rather a flag "has been cloned by google in last 24h" for each go module?

But since it looks like the proxy fleet is made of a large set of nodes and each one of them only keeps track of their own state, if sourcehut were to only keep one counter for the whole user-agent it would lead to a large number of nodes not being able to refresh their cache for potentially long periods of time (there's nothing that would guarantee for example that it's always a new node that gets to make the one request per hour).

That's precisely why it's so unfair to ask the sourcehut team to try to come up with a solution to this problem: it's the very design of the proxy that Google put together that is causing this issue in the first place. And the sourcehut team has no control over their design. And to add insult to injury: when the sourcehut team offers recommendations on how to improve their implementation, Google responds with "it's too much work".

And to add a cherry on top, the only reason a single user-agent counter is even a possibility to discuss is because Drew made the Go team put a unique user-agent on the proxy in the early stage of this issue report.

The Linux firewall is designed specifically to be able to track and count packets by IP address and other metedata, and can do so extremely efficiently.

Setting up an IP based rate limiting rule in nftables takes a few minutes at max, and I am not a professional sysadmin.

I like this because it feels like malicious compliance and it would allow Google to seemlessly continue working should Google figure out their caching situation. Plus it uses a rarely used HTTP status. Love it.

I have seen more 429, almost all from large cloud providers, a fair amount in the last 5 years. Never before then, I was used to making rate limiting an in-band part of the API explicitly. AWS S3 will give you 429s before giving you 503s when you hit it with a go program touching every object or every version in a giant bucket from a 56 core machine and a few hundred Go routine workers. It is a good signal: if you do exponential back off per worker on the 429s, you won’t see the 503s.

Reply on news.ycombinator.com