Hacker News Reader

> 2. Not all requests are created equal. 20 requests a second for the same static HTML file? No problem. But if you have, say, a release page for an open source project with binary download links for all past versions for multiple platforms, each one being a multi megabyte blob, and a scraper starts hitting these links, you will run into bandwidth problems very quickly, unless you live in a utopia where bandwidth is free.

All of this is (and should) cached on a cdn. You can go 1000 QPS on this in that config.

I think a person should be able to set up a small website on a niche topic without ever having to set up a CDN. Until very recently this was the case, so it's sad to see that simplicity go away purely to satisfy the data needs of shady scrapers.

Shouldn't such big blogs be put on something like CloudFlare R2 or BackBlaze or even S3 with their caching in front? Instead of having your server handle such file downloads.

Cache also cost money as well. Nothing is free.

>you live in a utopia where bandwidth is free.

It's called peering agreements and they are very common. There's a reason social media and sites like YouTube, Twitch, TikTok don't immediately go out of business. The bandwidth is free for most users.

there's only a handful of entities in the world that are capable of peering. Most people have to pay for their bandwidth.

It can be done by whoever is providing you internet connectivity. Not everywhere adds extra charges for bandwith.