Ask HN: How do RSS readers handle items missing pubDates? | Hacker News Reader

Ask HN: How do RSS readers handle items missing pubDates?

I'm writing an RSS reader, and I noticed that items often miss publication dates. Regardless, other readers seem to be able to gather them anyway, I'm guessing through the actual site.

Is this generally done through web scraping, archival information, web crawler data, or something else entirely?

I'm asking because I would hate to resort to those types of methods as they are generally heavy on resources and unstable.

View on HN

It's possible that the feed reader attaches it itself based on when it appears in the feed since the reader will keep polling the same feed over time

I’m not understanding your suggestion. It’s using Microsoft Syndication Feed to parse the RSS files which themselves are missing pubdate info. There’s nothing stored on disk.

Feed I'm parsing

http://rss.cnn.com/rss/cnn_topstories.rss

In this case most items have <pubDate>'s, for the ones without you use the feeds <pubDate> in the <channel>

Using the same channel pubdate for all the items with missing ones wouldn't be accurate.

I'm using The Old Reader as an example and all of their missing items have different dates.

just dates or date and time?

date and time

They are probably polling the feed.

What do you mean by this, they are listening for changes in the webpage?

They check the feed every x min, if a new item appears they use the current time (or the channel pupDate) as the item pubDate. There might be more accurate ways but this is good enough.

How is your thing suppose to work? Are you going to load all the feeds for every visitor? Even if it is just for personal use, how many feeds are we talking about? If you pull 20 per second and have only 200 the 10 seconds delay is going to be more annoying than it needs to be. You should probably consider storing the feeds on disk or in a db.

My goal is to load all feeds for every visitor, 200 feeds at first but ideally more.

Like another person suggested, I will likely use Internet archive to make an api request and retrieve date time. If that’s inaccurate then I will use polling, but that does seem expensive.

Are you suggesting storing the actual rss xml file in a database to be reused efficiently?

Store the feed item details in a database. That way, you can store the pubdate if one is present, and if not, just use the timestamp for when the item was first detected

When I've done it in the past, I've just attached the current timestamp of when the item's checked as the pubdate

It might be wrong, but it's usually pretty dang close to right (presuming the RSS feed isn't brand-new to the reader)

NetNewsWire uses [0] ‘dateArrived’ as a fallback option. As an example.

[0] https://github.com/Ranchero-Software/NetNewsWire/blob/941342...

Thanks, there doesn’t seem to be a description of dataArrived

Query the Internet Archive’s CDX server for this info. Consider caching the data for performance and good netizen reasons.

https://github.com/internetarchive/wayback/blob/master/wayba...

This is really nice, thank you very much!

There always is a date some place. Find or derive them, order them by how resource heavy and/or crappy they are. Chose wisely for the type of reader you are making and/or create configuration options for it.

Unfortunately, I think that's what I'm going to have to do.

no no, this is the fun of building the rss/atom reader!

My funniest moment was when I didn't notice someone deleted their feed and the missing page redirected to the front page. The many layers of failbacks just parsed the html as if it was a feed. It didn't have <item>s so it tried to parse it without items, it didn't have <link>s or <guide>s so it ended up looking for <a>'s in the <description> but it didn't have <description> so it just accepted any <a>'s and used whatever was in the <a>'s as the <title>.

I still think it will be fun in an inconsistent, man made jungle of the web sort of way. And I agree the types of mishaps are really interesting because they almost can't be replicated since they depend on so many environmental factors they almost appear physical.

There's an interview somewhere on Youtube of Bjarne Stroustroup, I think it was one of those Big Think videos. He was asked what he thought about AI and Machine learning, his response being that he's not interested in things that aren't sound.

That really stuck with me, I don't think web scraping is sound, it's guessing retrieval at broken fragments. I admired The Old Reader for including pictures in it's feed, then I saw a CNN post on there about he new M2 Macbook.....with a picture of a kitchen blender as the headline, missing the 5 or so images in the link of M2 Macbooks.

Id be surprised if many readers did something besides default to the current time if a date wasn't specified.

Looking at it from another point of view: should a RSS reader trust pubDate?

What better source is there?

You could just display the feed items sorted by the time you fetched them, optionally displaying an informative only date for the entries with pubDate.

Reply on news.ycombinator.com