Hacker News Reader

Comment by djhworld | Hacker News Reader

I've mentioned this on HN before, one pattern I've noticed is a lot of sites tend to truncate their RSS feed so you have to click through to see the full article etc. This is mostly OK but kinda frustrating, as in the before-covid times I used to ride the London Underground a lot where no celluar signal is available in the tunnels - so my offline RSS reader would be useless.

So I wrote a little web service accepts an RSS feed, and then crawls each link in it, parses the HTML to extract the text using one of those text extraction libraries, and then "republishes" a new version of the feed (with the full article) which I then subscribe to in my RSS reader so I can read the articles offline, e.g.

example.com/rss?feed=foo.net/artcles/rss

I've never open sourced it though because I guess it's a bit of a grey area - the sites want you to go to the full URL so they can show you ads etc

View on HN

I still heavily use RSS, but usually unsubscribe from feeds that truncate (unless it is a very long article).

It isn't a religious thing - the vast majority of the time, the truncation is way too short to properly advertise the content. The mystery doesn't move me to click-through, quite the opposite - it is annoying, so I remove the annoyance.

I understand the tradeoffs for folks who want to be paid via ads, but the utility just isn't there for me - I do not lack for reading options, and if something is really good, it'll find me some other way.

I personally truncate the RSS feed for my site, because it's mostly interactive experiences, and I would rather them experience the interaction. I don't have ads on my site though. I've been trying to come up with a good middle ground though.

I use Feedbin, and read Feedbin on my Mac and iPhone using the Reeder app. Both are able to extract the full text (automatically on a per site basis if wanted) of most sites without issues.

I guess they just follow the URL, extract the body text the same way that sites like Instapaper do it, and give that back.

Probably more RSS readers can do the same.

I've worked on something similar for some years: http://ftr.fivefilters.org

An older version available on BitBucket: https://bitbucket.org/fivefilters/full-text-rss/src/master/

I'm a huge fan. Here's my Docker-ized take: https://github.com/alrs/full-text-rss-docker

EDIT: full-text-rss + newsboat + nyt screenshot: https://lsngl.us/@alrs/105652475456016345

Oh that's fantastic! I've been meaning to look into providing a Docker version at some point, will definitely look more at how you've done things here.

clicking through is what killed it for me too. I just never went through the trouble of setting up crawlers and stuff. Looks like open source is to the recue nowadays. I am pretty sure there was nothing as easy as that back in the day. Thanks for sharing this!

Are you also behind the other fivefilters.org tools? I use Push to Kindle often and I've paid for the Android version. Thanks!

Thank you. Happy to hear it's been useful for you. And yes, I work on Push to Kindle too.

Newsblur has a setting per-site which allows you to have it use either the feed body or attempt to pull in the full text. When I was commuting on the subway (prior to ubiquitous underground coverage) here in DC, I used that heavily. One nice thing is that downloading it on your client avoids the site having to get in the middle of the copyright concerns about replicating their servers.

nice! I've been using newsblur for a few years now but I've never delved into the settings. is this option only available on the web version? i can't see anything on the android app

It’s in the site settings menu on the website. I haven’t used the Android app in years, so I can’t help there.

I have used both of those methods when publishing... I do hesitate on the full article approach, but I don't run ads. (And to be clear, I do provide a whole-article feed)

One thing I don't like about the whole article RSS is that I get requests to narrate my whole website or blog experience, in addition to the normal writing about stuff.

In effect the full-article RSS audience has its own experiential needs, which include "tell me about what you added to the non-blog sections, tell me about the new tools in the sidebar, or the new article footer format, or even hey I never heard about the new design," and so on. Sometimes I also alter the CSS for a single post, when it makes sense.

Some of this narration is just expected if your main format is traditional, plain blog. But personally my blogs tend to evolve and incorporate other stuff. And for sites with even more of an experiential aspect than my own, I can see why RSS could drive the publishers a bit mad, increased exposure or no.

Because of this experience, personally I would rather try reading via a page-download tool or format, even automated w3m or something, and see if that's an improvement. You'd get the full article in plain text, but you'd also get some of the other stuff around the site, including links which you may even be able to mark for later download.

This would largely have saved RSS for me. I'd argue there was a glorious era in late 2000s where you could almost replace your browser with a good feed reader and get vast majority of news/info this way. It was just so nice to have a single UI surface for all news stories rather than poking through many different websites.

I understand my using the sites that way contributes nothing really to their bottom line, I never saw any ads, but that of course was half the appeal. The early days of "web2.0" people often just naively dumped everything in the RSS feed.

There are a few services that get full text from RSS feed items for you

I still find feeds from truncated useful as I can scan through the headlines quickly; I rarely want to read everything a publisher puts out there. When something catches my eye my RSS viewer, Feedbin, can attempt to fetch the entire one for me and if that fails visiting the site directly is tolerable.

A lot of web pages are just completely unusable or unreadable now.

It uses all sorts of manipulations to auto play video ads, pop ups, picture unders, user tracking, page interaction telemetry, etc.

All kinds of nonsense to display an article to you, which if it really is important, then another writer would have also written about it.

Some websites I just avoid completely, when I see the url.

These websites should just go with a pay-to-read subscription model instead. If their junk is interesting enough, then they’ll find enough people willing to pay a few cents to read it.

> I've never open sourced it though because I guess it's a bit of a grey area

For whatever it's worth, I do click-through RSS feeds on my blogs just because I was too lazy to set up full text feeds and figure out how stuff like video would work and whether or not it would be a better end-user experience to alter the content in some way on style-heavy pages. Right now I'm constructing the feed myself. But if someone ever wanted to use a tool like you're describing on my blogs, I would have no objection. Don't feel guilty about scraping a site I build or building tools to do so, if anything it would make me feel less guilty about never getting around to setting up full-text RSS feeds. :)

I don't run ads or analytics anyway, but even if I did, as far as I'm concerned anyone who has the right to visit a page also has the right to download it, and to use automated tools to download it on their behalf[0].

I don't see any moral difference between using an adblocker and scraping a web page. Both are hitting the same endpoints and selectively displaying content that the user wants to see.

[0]: https://anewdigitalmanifesto.com/#right-to-delegate

> I've never open sourced it though because I guess it's a bit of a grey area - the sites want you to go to the full URL so they can show you ads etc

I honestly would not mind much if sites just included ads in their .RSS feed? Some bloggers I follow have their rss 'sponsored' and will do a sponsor post once a week. If ad insertion is really so important a site could just have labeled ads as part of the content stream. It's not ideal, but I'd prefer it to a feed that only has 2 or 3 teaser sentences and then forces me to click through.

I actually took Wired and Slate off my RSS feed just because I got annoyed at how often I would click through and get a paywall. Usually I was using up my free articles just to see what the article was about without reading it.

Some RSS readers implement something similar. I currently use nextcloud news, that does this: https://github.com/nextcloud/news/pull/563

Basically, for feeds where I have "full text" enabled, it fetches the article page with mozilla's "reader mode" implementation and extracts the full text.

It isn't really usable with aggregators such as planet kde/gnome, though.

I thought the whole point of RSS was as an index to new content. It never occurred to me to put the whole article in the feed... surely a good summary and a link gives the consumer the choice to load the article or not, rather than having to load all the articles all the time?

> I've never open sourced it though because I guess it's a bit of a grey area

I'm maintaining the TT-RSS plugin feediron https://github.com/feediron/ttrss_plugin-feediron that fetches full-text data, so my thinking is this:

At the end of the day if it's a openly available website and you are personally (through your own server) fetching the resources I don't think anyone has a right to complain.

Now if you were offering it as a service it might arguably be a bit more grey, but only if you're ignoring the robots.txt file

I think this kind of service is actually harmful to content providers. They can not benefit from the content they created if readers always consume it on the RSS readers. That's one of the reasons many blogger/website don't provide RSS feeds anymore.

The RSS aggregators should encourage readers to go to the original website. So when I got the chance to build my own RSS reader(https://feeds.pub) I only shows a title and link of an article to readers to drive them to the original website.

Yeah I always go to the site regardless. I always use list view so all I get is a title and an excerpt.

Do you allow for a feeds list format? In other words, can I just see a list of feeds and folders, and when I click on the feed, it will display a list of articles from that feed? Or is it like Twitter, where everything is squished together.

That's an interesting use case. My POV has always been that the site itself is the best representation of the content and completely ignore the feed content.

For me it also depends on the content. Some sites I'd rather just read from in the feedreader such as a news site, but for things like YouTube videos I would rather watch on YouTube than watch via the embedded player.

However I find news sites are the most common offenders of the truncating which I guess I can understand.

Interesting.

I am not sure if there are many doing something similar to mine. My pattern is to Open all my interested Links in New Tabs. That is why I thought RSS Reader should always be within a Browser or as Web Services like Feedly.

I do offline Reading for long articles only. And Saved via Reading List on Safari that does automatic downloading. But it is quite problematic for me because most of the time I need to look up something related while I am reading on things that I dont understand or assumed wrongly. And this is still best done on a PC with Browser.

So I now only listen to Music, Reading Novel or WebComic / Manga for offline.

I used to live in London (8 years ago) and I did exactly this because of the Jubilee line. I had written a perl program rss.pl that I would give several URLs to generate custom RSS feeds for those websites that provided poor, summary only, feeds. I had it running in a crontab and just posted the feeds to my own webserver. This brings back memories. I live in Tokyo now where there is internet access everywhere so I retired my perl program.

One of the reasons I went from placing a full article in my RSS feed to truncating it into a descriptor and leaving the link as the "source of truth" was I found that there a bunch of intentionally crippled readers, like Thunderbird, that strip all the markup and turn my article into an unreadable mess.

Traffic to both the feed and to the main site increased noticeably when I reduced it to a descriptor.

> a lot of sites tend to truncate their RSS feed so you have to click through to see the full article

This is unacceptable to me, and if a site does this I unsubscribe immediately.

Great tool you have! I was considering building something like this (but directly in a feed reader), by any chance do you intend to release the source for it so that people can run it themselves?

Can you share which libraries you’ve had success with for the content extraction? I’ve always wanted to play around with one

> I've never open sourced it though because I guess it's a bit of a grey area - the sites want you to go to the full URL so they can show you ads etc

Another reason, which I think is fair, is that with full articles in the RSS feed, the feed can quickly become a fairly large chunk of data.

On the one hand, I hate that as a user. On the other hand, I'd love to know how many people actually read a story.

I haven't found a good solution, but I just suck it up and serve the full text anyways

This exact problem applies to podcast analytics. "Downloads" is only a proxy for "listens" / ad impressions.

It seems to me like there needs to be a standard pingback endpoint (either in /.well-known or, better, right in the podcast/rss XML) that your client pings when it seems like you've consumed the content (scrolled through most of a web page, listened to 80+% of a podcast), but only if you've opted in to that.

Out of curiosity, did you manage to implement auto crawler that can always find the correct html element for any site and remove rest? Or did you have to create per-site rules for it?

I never subscribe to feeds like that, unless I guess I really care about the source, but it hasn't happened so far.

I don't think it's a grey area. They could show you ads in the HTML, they choose not to, and instead choose to let outside companies execute arbitrary code on your machine to track you. "Sites want you to go to the full URL so someone else can track your web use" sounds way less innocent and, to me, downright hostile. No one should feel morally obligated to allow that to happen.

Hugo, and other static site generators that support RSS, can easily be modified to provide the full content.

Miniflux does something similar

I'd like that. This pattern you mention bugs me.

This is literally the only reason I can see somebody using this pattern. The similarities with a paywall are remarkable.

Reply on news.ycombinator.com