Hacker News Reader

Comment by Havoc | Hacker News Reader

Blocked for 2 hits in 20 minutes on a light protocol like rss?

That seems hilariously aggressive to me, but her server her rules I guess.

If your feed reader is refreshing every 20 minutes for a blog that is updated daily, nearly 99% of the data sent is identical. It looks like Rachel's blog is updated (roughly) weekly, so that jumps to 99.8%. It's not the least efficient thing in the world of computers, but it is definitely incurring unnecessary costs.

I opened the xml file she provides in the blog and it seems very long but okay. Then I decided it is a good blog to subscribe so I went and tried to add to my freshrss selfhosted instance (same ip obviously) and I couldn't because I got blocked/rate limited. So yes it is aggressive for different reasons.

This post (before reading your comment) actually made me look into my own freshrss setup.

NixOS defaults to refresh frequency of every 5 minutes[0] (0_0).

I had noticed some blogs blackholing me before, but never quite made the connection.

So now it is configured to fetch every 12 hours. I believe that is fair.

[0] https://github.com/NixOS/nixpkgs/blob/d70bd19e0a38ad4790d391...

Same, I made 3 requests in total and got blocked.

I know she's mentioned this particular problem before on her blog, I don't remember where to find it offhand now but my vague recollection is that because browsers have largely removed the ability to directly view RSS feeds she doesn't consider this a significant issue anymore.

Why did you view the XML file directly?

> Why did you view the XML file directly?

There are many reasons why I do personally this.

1- Check that the link actually loads and works!

2- See how much content and does it contain last n or all feed history by default

3- To see if the feed gives summary or full content of posts

4- Just for curiosity like in this case I wanted to see what is this feed that prompted a blog post that reached HN front page.

It is usually a superposition state of those reasons. But this is why it is aggressive limit and I know it her server her rulee but this wasn't pleasant experience for me as an end user. I was just sharing my experience.

I feel like a reasonable way to deal with this situation might be to look at the user agent: if two requests come from the same IP but different user agents, then it's likely that it's either actually two completely different people (behind a NAT), or this situation the GP described.

That's certainly a bit more effort to implement, though, and the author night not think it's worth the time.

Weird. Those should have had different user-agents, and I would guess it cannot be purely based on up.

Yeah, that's insane. Pretty much telling me not to subscribe to your blog at that point. Like sites that have an rss feed yet put Cloudflare protection in front of it...

The correct thing to do here is put a caching layer in front so that every feed reader isn't simultaneously hitting the origin for the same content. IP banning is the wrong approach. (Even if it's only a temporary block, that's going to cause my reader to show an error and is entirely unnecessary.)

It should be a timeboxed block if anything. Most RSS users are actual readers and expecting them to spend lots of time figuring out why clicking "refresh" twice on their RSS app got them blocked is totally unreasonable. I've got my feeds set up to refresh every hour. Considering the small number of people still using RSS and how lightweight it is, it's not bad enough to freak out over. At some point all Rachel's complaining and investigating will be more work than her simply interacting directly with the makers of the various readers that cause the most traffic.

Her rss feed is last 100 posts with full content.

So it means 30 months of blog posts content in single request.

Sending 0.5MB in single rss request is more crime than those 2 hits in 20 minutes.

I generally agree here.

There are a lot of very valid use cases where defaulting to deny for an entire 24 hour cycle after a single request is incredible frustrating for your downstream users (shared IP at my university means I will never get a non-429 response... And God help me if I'm testing new RSS readers...)

It's her server, so do as you please, I guess. But it's a hilariously hostile response compared to just returning less data.

> But it's a hilariously hostile response compared to just returning less data.

So provide a poor service to everyone, because some people doesn't know how to behave. That sees like an even worse response.

Send only one year's recent posts and you've reduced bandwidth by 50%.

People don't want to have to customize refresh rates on a per-feed basis. Perhaps the RSS or Atom standards need to support importing the recommended refresh rate automatically.

They don't need to change the refresh rate, though. They need to make conditional requests with an etag or a last-modified date, so the server can respond with a 304 not modified if no changes have been made.

No standards need to be updated. The client software needs to be a better HTTP citizen.

What about people who reside in the same place who have multiple RSS aggregators that scrape the same RSS? Her analysis will not handle that I think. At some point she is going to have to talk to the engineers that made it if she wants something done. Or she could take it upon herself to fix the software (at least the ones that are open-source). If she's just sharing the investigation then it's fine. But if the goal is to get the problems fixed, whining to us is probably the least efficient way to do it. She is knowledgeable enough to fix probably half of the RSS readers that she is complaining about and definitely knowledgeable enough to engage with all of them about fixing their code.

If there were a widely supported standard for pagination in RSS, then it would make sense to limit the number of posts. As there isn't, sending 500kB seems eminently reasonable, and RSS readers that send conditional requests are fine.

"Pagination in feeds like ATOM and RSS?" - https://stackoverflow.com/questions/1301392/pagination-in-fe...

Sounds like something that could be scored in the rss reader tests.

Did you actually write 500KB as 0.5MB to make it sound BIGGER?

Clever.

Yes that's right. Most blogs that are popular enough to have this problem send you the last 10 post titles and links or something. THAT is why people refresh every hour, so they don't miss out.

I hate RSS feeds that don't include full content.

If only there were some kind of HTTP headers that could help them stop doing a GET every hour!

Gosh darn, if only I could say "Hey, please only send me the data if it's been modified since I last requested it an hour ago" somehow.

Sure, but whining to the broader public about it before talking to the engineers who made the offending software seems like a bad idea. The public doesn't really care and will keep using their preferred readers. In all my years on the Internet (a lot) I have never seen anyone complain about the volume of RSS traffic they got. If Rachel enjoys sharing her experience of investigating this issue, that's fine. But if she is sharing it with the expectation that her readers will randomly go fix other people's software, that's becoming unreasonable.

Complains about traffic, sends 0.5mb of everything.

That’s my kind of humor.

sigh feed readers set the If-Modified-Since header so that the feed is only resent when there are new items.

> Blocked for 2 hits in 20 minutes on a light protocol like rss?

I might be getting old, but 500KB in a single response doesn't feel "light" to me.

Yes, this is a very poorly designed RSS feed.

500KB is horrible for RSS.

It's reasonable to have whole articles in RSS, of you aren't trying to show ads or similar.

Whole articles are reasonable.

100 articles are not reasonable.

100 articles where most of them are 1+ year old is madness.

RSS is not an archive of the entire website.

Well, its note the entire website, and i find the "last 100 articles" rule way better than "last 3" or "last 90 days" (which some times is 0 or 1).

The host is fine with sending 0.5 MiB once (the client should be aswell from both a bandwidth and storage point of view).

The host is not fine with sending 0.5 MiB every 20 minutes, which could be easily avoided if the client would use the mentioned "If-Modified-Since header".

Thanks for being someone who actually knows this stuff among all the Dunning Kruger replies

> RSS is not an archive of the entire website

Whole-article feeds end up become exactly that - a local archive of a blog.

I believe if you read carefully, it's not blocked, it's rate limited to once daily, with very clear remediation steps included in the response.

If you understand what rate limiting is, you block them for a period of time. Let's stop being pedantic here.

72 requests per day is nothing and acting like it's mayhem is a bit silly. And for a lot of people would result in them getting possible news slower. Sure OP won't publish that often but their rate limiting is an edge case and should be treated as such. If they're blocked until the next day and nothing gets updated then the only person harmed is OP for being overly bothered by their HTTP logs.

Sure it's their server and they can do whatever they want. But all this does is hurts the people trying to reach their blog.

72 requests per day _per user with a naive feed reader_. This is a small personal blog with no ads that OP is self-hosting on her own hardware, so blocking all this junk traffic is probably saving her money. Plus she's calling attention to how feed readers can be improved!

My reason for smacking stuff down is that I don't want to see it in my logs. That simple.

Even if they had 1000 feed readers which would be a massive amount for a blog, if you can't scale that cheaply, that's on you.

As I pointed out, her blog and rate limiting are an extreme edge case, it would be silly for anyone to put effort into changing their feed reader for a single small blog. It's bad product management.

Of course she can. It's static. She doesn't want and I understand. She's signaling their clients an standard call to say "I think you already have read this, at lest ask me first when this changed the last time".

If you choose to run a poorly implemented rss feed and not scale it cheaply you lose any sympathy from me.

So long as you know what If-Modified-Since is, and use it, you can have all or none of the sympathy you want.

> 72 requests per day is nothing and acting like it's mayhem is a bit silly.

72 requests per day per IP over how many IPs? When you start multiplying numbers together they can get big.

I invite you to run your own popular blog on your own hardware and pay for the costs. It sounds like you don't know what the true costs are.

I do run a popular blog, and a $5 a month Digital Ocean droplet handles millions of requests per month without breaking a sweat.

If every user is collecting 36mb a day like in the story here, your droplet wouldn’t even be capable of serving 500 users a month without hitting your bandwidth limit. With their current rates, your one million requests would cost you around 10 million USD.

30 * 500 * 36mb = 560gb and I have 1tb a month on my apparently $6 droplet

Correction - from my billing page it's $4.50 a month, from the resize page it is $6 so I'm guessing I am grandfathered in to some older pricing

That's ridiculously big quantity of data to serve a seldomly updated blog just because the client doesn't want (or know how, or think about) to implement an easy and old http method.

Imagine the petabytes of data transferred through the internet saved if a couple RSS clients added that method.

If OP enabled gzip then this 36mb would be 13mb.

If OP reduced 30 months of posts in rss to 12 months then this 13mb would be 5mb a day.

Using Cloudflare free plan and this static content is cached without any problem.

Yeah, but also... if RSS readers behaved correctly, it would be 512 kb. (170 kb with gzip, if she didn't enable it like you imply – I'm too lazy to check, but I assumed it was on.)

I think making clients behave correctly is much more sustainable solution, although we could do better than doing so at the cost of the end users.

This entire thread is a vivid illustration of why software is so shitty in general these days.

OP has never said that this is about financial aspects of things.

Yews, it's about enforcing their preference on how others should interact with OP's published site feed, on principle. Which is always an uphill battle.

It's about enforcing that people follow standards. Which is still an uphill battle, but at least it's based in something sane. Their work on this has resulted in improvements to a whole slew of popular feed readers that should make life easier for a chunk of the internet, not just OP's own site.

I serve 30tb/month for $30/mo on my own colocated hw

Sounds like you don't know how to scale for cheap.

And since I've ran integrations that connected over 500 companies. I know what a rouge client actually looks like and 72 requests per day and I wouldn't even notice.

Good on you champ.

More like a skill issue or just decision to make your life more difficult.

It is free and easy to scale this kind of text based blog.

> More like a skill issue

Hey, I think you mistook HN for Reddit.

But it's not a "light" protocol when you're serving 36MB per day, when 500KB would suffice. RSS/Atom is light weight, if clients play by the rules. This could also have been a news website, imagine how much traffic would be dedicated to pointless transfers of unchanged data. Traffic isn't free.

A similar problem arise from the increase in AI scraper activities. Talking to other SREs the problem seems pretty wide spread. AI companies will just hoover up data, but revisit so frequently and aggressively that it's starting to affect the transit feeds for popular websites. Frequently user-agents wouldn't be set to something unique, or deliberately hidden, and traffic originates from AWS, making it hard to target individual bad actors. Fair enough that you're scraping websites, that's part of the game when your online, but when your industry starts to affect transit feeds, then we need to talk compensation.

That’s a bit disingenuous. 429s aren’t “blocking”, they’re telling the requester that they’re done too many requests and to try again later (with a value in the header). I assume the author configured this because they know how often the site is going to change typically. That the web server eventually stops responding if the client ignores requests isn’t that surprising, but I doubt it was configured directly too.

Semantics. 429 is an error code. Rate limiting...blocking...too many requests...ignoring...call it whatever you like but it amounts to the same, namingly server isn't serving the requested content.

> 429s aren’t “blocking”

Like how "unlimited traffic, but will slow down to 1bps if you use more than 100gb in a month" is technically "unlimited traffic".

But for all intents and purposes, it's limited. And 429 are blocking. They include a hint towards the reason why you are blocked and when the block might expire (retry-after doesn't promise that you'll be successful if you wait), but besides that, what's the different compared to 403?

I would disagree. Blocking typically implies permanence (without more action by the blockee), and since 429 isn’t usually a permanent error code I wouldn’t call it blocking. Same applies with 403, it’s only permanent if the requester doesn’t authorize correctly.

I would say it's disingenuous to claim sending HTTP status and body that is not expected for a period of time is not blocking them for that period of time. You can be pedantic and claim "but they can still access the server" but in reality that client is blocked for a period of time.

In that case, I should be irate that the AWS API blocks me many times per day. Run `aws cli service some-paginated-thing` and see how many retries you get during normal, routine operation.

But I’m not, because they’re not blocking me. They’re asking my client to slow down. Neither AWS nor Rachel’s blog owes me unlimited requests per unit time, and neither have “blocked” me when I violate they policies.

They literally do block you for a period of time until you are out of the rate limit. That is how rate limits work. That's why you don't get to access the resource you requested, because their system literally blocked you from doing so.

See when you're trying to be pedantic and all about semantics, you should make sure you've crossed your Ts and dotted your Is.

> Block – AWS WAF blocks the request and applies any custom blocking behavior that you've defined.

from https://docs.aws.amazon.com/waf/latest/developerguide/waf-ru...

And my favourite

> Rate limiting blocks users, bots, or applications that are over-using or abusing a web property. Rate limiting can stop certain kinds of bot attacks.

From CloudFlare's explainer https://www.cloudflare.com/learning/bots/what-is-rate-limiti...

Every documentation on rate limit will include the word block. Because that's what you do, you allow access for a specific amount of requests and then block those that go over.

Reply on news.ycombinator.com