Hacker News Reader

Comment by derekzhouzhen | Hacker News Reader

A feed is not a block chain; the feed publishers are free to do anything on their feeds. So the feed readers must be able to protect themselves from serving deleted items in perpetuity. I ran a server-side feed collector in roastidio.us for airss.roastidio.us users, my algorithm is very simple:

I keep only the top N items in the latest version of the feed; everything else are dropped. For a fast moving feed, if the users don't read soon enough some items will be missed. This is still better (in terms of usability) than the bombardment of items to a user just coming back from a long vacation.

View on HN

Different conversion for a different topic.

> For a fast moving feed, if the users don't read soon enough some items will be missed.

This may work well for you but definitely isn't what I prefer. For example I often subscribe to recording feeds for conferences or other events. These feeds don't post for most of the year but them might have hundreds of entries. I definitely don't want my feed reader to drop all but the last 10 (YouTube for example keeps much less than the latest 100 videos in the feed). I want it to keep them all so that I can work through the list and watch the ones I am interested in.

So while your some solution might work in some cases this definitely isn't what I would want personally and in my opinion what Feedly did here was reasonable as a general behavior, even if it isn't optimal for this particular case. I'd rather have it handle the happy common case we'll then to hurt it for the rare case when someone makes a mistake, especially when the downsides for the mistake aren't that high.

> I want it to keep them all so that I can work through the list and watch the ones I am interested in.

Your usage case can be better served by visiting the website directly, after you learn that there are some updates from the feed. There is no guaranty that a feed aggregator would not miss anything anyway; this is the the nature of the poll based delivery.

Thanks for explaining my use case to me.

- How to I remember where I am when I have watched half of the videos?

- What if two events happen around the same time. Now I need to check both instead of them being in the same place.

It turns out that leaving items in my feed reader and dismissing them as I watch or decide not to watch works incredibly well.

You are using the feed to overcome the usability problem of a poorly designed website. I have seen my fair share of poorly designed websites so I can't argue with that :)

My feed reader's intended use case is to following a large number of slow moving feeds like personal blogs, which are usually not bursty and not super important if a few items fell in the crack.

Maybe, but I don't expect most websites to track which videos I have watched and I don't want to need to create an account or somehow sync my watch list across devices. So I just use my feed reader for that and it works excellently.

> My feed reader's intended use case is to following a large number of slow moving feeds like personal blogs

This is also the majority of my feeds. While I agree that a few items falling through the cracks is a big deal I would still prefer if they don't.

But I think the morale of the story is that there is a huge set of ways that people use RSS Feeds. They are a generic system for distributing updates and it is hard to make quick fixes that work well for all of the wide variety of use cases that people have.

You're just picking from different trade-offs, not doing something better nor worse (nor right nor wrong).

A major use-case of RSS readers is to have a local, eternal copy of a producer's content to read at your own leisure. I have content in my RSS reader that I cherish from blogs that aren't even online anymore. That I don't need to visit poorly designed websites is only a tertiary perk—I don't need to visit the origin, ever.

There's not much to argue here, though. It's just a different offering. For example, I use https://sumi.news/ (created by some HNer) and certainly don't use it read the entire history of every news website. Nothing wrong with that.

> You're just picking from different trade-offs

Of course. I am not feedly and I can't afford to keep everything for all the feeds ever subscribed by my users in my database. There are feeds that keep thousands of items and there are feeds that have broken date fields so I cannot prune by date reliably.

> I keep only the top N items in the latest version of the feed; everything else are dropped. For a fast moving feed, if the users don't read soon enough some items will be missed.

I completely dislike this behavior. I saw this on Ars Technica some years ago, where it was said that only the most recent 20 items (?) would be in the feed. Ars can be a fast changing site, and since full text feeds are a paid privilege, I thought it was a disservice to paying users not to have the ability not to miss out on something just because they were busy with something else for a day or two.

I don't have the time to read everything. I'd rather miss something from the busy news site than miss something from independent bloggers.

The problem is that RSS doesn't really have a concept of "deleted". In general feeds just contain the newest items. Items not on the feed are generally considered old, not deleted.

You can play with heuristics like "This feed is usually sorted by published date and these entries I saw previously should still be here but are missing, maybe they were deleted" but not even have published or updated dates and they aren't always accurate.

This gets even more complicated with WebSub which allows hubs go remove entries they have already seen. So now you need to not do deletion on those requests, and if you turn down your poll rate for feeds with WebSub (which Feedly does IIRC) you might not even fetch the full feed to notice that items are missing for weeks.

The one thing that would actually be reliable here is probably Atom's pagnation and archiving extensions which you can follow to get a view of all of the items in the feed and prune missing ones. However you probably don't want to do this every fetch (otherwise it kinda defeats the point of pagnation). Maybe you could combine this with other heuristics to come up with a decent system. But I'm not even sure these have ordering guarantees so I'm not sure if you are going to fetch potentially unlimited pages just to check if an item was deleted.

If we did want to solve this problem the best option would probably be to add a tombstone element to the feed to explicitly remove items.

TL;DR as far as I am aware neither RSS or Atom have the concept of deletion. Much like sending someone 100 email newsletters unrelated to their subscription will piss them off with no recourse try not to send them 100 feed updates that they don't want.

Edit: It looks like there are tombstone proposals for both RSS and Atom: https://oleb.net/blog/2015/11/rss-feed-item-deletions/. Maybe the author could have tried this, but IDK if anyone supports them.

Reply on news.ycombinator.com