RssHub – A feed aggregator that can generate feeds from pretty much everything | Hacker News Reader

RssHub – A feed aggregator that can generate feeds from pretty much everything

docs.rsshub.app

235 pointstvvocold60 comments

How come noone has managed to crack the RSS rock yet? Everything has an RSS feed (though it's decreasing). Rss readers have more content than any social network. It seems like you should be able to build something with this that has broad adoption. There has to be something missing.

My theory has been for a while that what is missing is publishing. People don't just want to consume, they also want to share what they find and like and what they made themselves. I even started working on a web-based rss-reader/(micro)blogging thing at one point. Maybe I should give that another crack.

Isn't ActivityPub supposed to fill the gap?

Yeah its predecessor, OStatus, was pretty much that: you start with a feed of anything, then you add another feed for the reader to comment, then a way to send the comment back to the source so everyone can see it, and lo and behold you've got yourself a social network brewing. Activitypub is a rethinking of the whole thing with clear requirements and specs instead of some informal pieces added here and there, which is a good thing. Except all the existing RSS feeds can't be used as such.

If you don't know it, take a look at Shaarli[1], it's a bookmarking web app that allows you to share your bookmarks via RSS, you can add tags and comments, but you can also have "notes" without a link that can be used as microblogging posts.

It's a project from Sebsauvage[2] (an old school french blogger), who wanted to build a social network around the RSS technology.

I know there are projects around it to aggregate multiple shaarli and use it as an actual social network based around sharing links with RSS. I didn't search about it, but it seems that shaarlo[3] is one of them. EDIT: actually, projects like this are listed in shaarli's documentation[4].

EDIT2: tt-rss-shaarli seems to be what you want, Tiny-Tiny RSS with Shaarli, an aggregator with a sharing/commenting functionality.

[1] https://github.com/shaarli/Shaarli

[2] http://sebsauvage.net/wiki/doku.php?id=php:shaarli

[3] https://github.com/DMeloni/shaarlo

[4] https://shaarli.readthedocs.io/en/master/Community-&-Related...

This is exactly what I don't understand about all the feed readers out there. It seems so obvious that the next step for feed readers would be social interactions like sharing, also it would be to easy to add I guess.

Additionally the aspect of consuming YouTube via RSS feed. Feed readers could simply add small logic which transforms a given YouTube account into one of

  https://www.youtube.com/feeds/videos.xml?channel_id=<account_id>
  https://www.youtube.com/feeds/videos.xml?user=<account_name>

And so on for similar services. RSS has such great potential but somehow nobody uses all of it.

Is there anything I miss? Why don't feed readers support external services (like the YouTube example from above)? Why don't feed readers add social features?

You can add the RSS feed of a youtube channel with the Tiny Tiny RSS bookmarklet, it's working for most channels, but not all for a reason I don't understand yet.

YouTube channel feeds, if they have one, are at: www.youtube.com/feeds/videos.xml?channel_id={channel_id} Some channels have a custom url and the id isn't obvious but it should be in the page source somewhere.

For example, "It's Okay To Be Smart" is at "www.youtube.com/user/itsokaytobesmart". Right click, view page source, search for "/channel/". That finds "<link rel="canonical" href="https://www.youtube.com/channel/UCH4BNI0-FOK2dMXoFtViWHw">". So the feed url is:

    https://www.youtube.com/feeds/videos.xml?channel_id=UCH4BNI0-FOK2dMXoFtViWHw

I'm not sure what the relationship between channels and users is. I think it's many-to-many and I'm aware of 3 feed types:

    https://www.youtube.com/feeds/videos.xml?user=<username>  
    https://www.youtube.com/feeds/videos.xml?channel_id=<channel id>  
    https://www.youtube.com/feeds/videos.xml?playlist_id=<playlist id>

Would love to hear if there are more.

Woah, I spent more time than I care to admit trying to find some feeds. I thought I had the secret sauce all figured out :) https://www.youtube.com/feeds/videos.xml?user=itsokaytobesma... is much easier to use. I can think of a few channels where the play list thing will come in handy if it works. Gamers where I only want to see 1 out of 50 games they play and are who are orginized enough to keep seperate playlist is one example. You can't even do that with YouTubes own subscription method as far as I know.

Feedly?

Because RSS gives readers too much signal, and not enough noise (ads, profiling), and doesn't keep users within the walled garden ecosystems. That's why apps are popular, too. It is all about control. Another disadvantage is that RSS doesn't have native incremental updates, which increases data volume. Though that isn't very relevant in 2018.

As an RSS lover, I have zero interest in social features. If I occasionally want to share something, I can email/WhatsApp/Slack the URL; but for the most part I don't, and I'm able to read without the distraction and noise.

RSS is used by those who aren't into "social" addiction, but just want a quick listing of relevant news from a super-simple news feed (as in "really simple syndication"). A couple years ago Google embraced, extended and almost extinguished RSS, so I'm not sure RSS users are willing to obtain/share their feed URLs from a pointless central service; just go to the web sites you're reading frequently, check if they have an RSS feed announced in their HTML metadata (as indicated by an RSS symbol in eg. FF), and add the URL to your feeds.

Simple:

- most RSS readers suck so bad you can't even retrieve years old articles. So if you follow blogs that don't post often, you simply can NOT get any of their older posts. Imagine a blog that has one seriously good article once a month. Typically you can't get more than a year's worth of posts, so that's like 12 articles. Why?

- the best RSS app I had on iPad let me archive feeds for a LONG time. I used it as a kind of a library. Returned to articles many times (philosophy and such). Most RSS readers out there have ZERO functionality in terms of archiving content indefinitely. (meaning : they typically have some leeway in max articles stored, or how old to archive feeds.. but you know you simply can not realy on the app to save your favorite articels, at some point they will be deleted).

- inability to highlight text. Again, if I read something I want to be able to at least highlight some text. Most RSS readers don't do this.

Instead RSS readers do the same thing every other RSS reader does. Basically, it's like the developers of RSS apps think that the only purpose of retrieving articles is for fueling some kind of "feed" addiction. Like the only purpose of content on the web is to satisfy an urge to quick scan and move to the nex article.

It's like the devs of RSS apps are so drugged out themselves in this day and age of short attention span, that they can't envision other uses for a feed reader than satisfying this urge for the daily/hourly doses of "novelty". Click click click. Swipe swipe swipe.

/rant off

You're free to develop an RSS reader with those features yourself. RSS is - on paper - an extremely simple protocol, but since many feeds are broken (not wellformed/improperly escaped XML, often incompetently copied from HTML, with spam, etc. etc.), RSS readers must integrate individual feeds almost on a case-by-case basis, with feed-specific fixes and heuristics. And there goes your project budget. Considering RSS isn't exactly growing (because the number of valuable news site isn't neither), the state of RSS is as good or bad as it always has been. I don't know if that is actually a good thing, because when big media (Google) last turned their eye towards it, it monopolized and almost ruined it.

You're free to develop an RSS reader

"Free to" != "Capable of"

It's that kind of attitude that keeps Linux from going mainstream, too.

This is a user complaining about user issues, and the response is "do it yourself." It is both not helpful, and condescending.

> - most RSS readers suck so bad you can't even retrieve years old articles. So if you follow blogs that don't post often, you simply can NOT get any of their older posts. Imagine a blog that has one seriously good article once a month. Typically you can't get more than a year's worth of posts, so that's like 12 articles. Why?

Most blogs don't include more than the last ten or so articles in their feed. Fetching backlogs of posts is beyond the scope of RSS. Unless you're complaining that posts you've already downloaded aren't persisted indefinitely?

This seems like the sort of thing that should be in the spec for a feed. RSS is basically a textual representation of a site's available articles; you have to explicitly GET the feed to see any new posts (at least without PubSub). If you're going to provide a static xml representation of your site's content, there's no obvious reason why you wouldn't want that to extend back indefinitely.

Something like feed.xml?page=2 seems fairly sane.

By looking into Podlove[1] I stumbled onto a specification[2] by the IETF for just that. It doesn't seem to have much support though, neither on the feed creator, nor the client side.

[1] https://podlove.org/paged-feeds/

[2] https://tools.ietf.org/html/rfc5005

https://bazqux.com/ has long (I think indefinite?) history, text highlighting, a starring system and built-in integration with Pocket/Pinboard/Evernote/whatever.

Keep in mind your history issue is often the fault of the feed, not the reader. If you subscribe to a new blog that only has a year's worth of posts in its feed, then that's what you get. The reader can't proactively follow every blog in the world just in case someone subscribes in 3 years and wants to see the full history.

I was so happy at seeing telegram, but it requires admin rights to the channel. There is so much content (a lot of it non-english) hidden away in telegram channels that I want to be part of the open web.

If someone wants to take it up, it is definitely doable with a telegram user account, but the APIs are not readily usable: https://github.com/captn3m0/ideas#telegram-to-rss

Also, if someone is looking to follow lots of GitHub projects using their release feeds, I wrote a OPML generator that uses your Starred repos to generate a OPML file for you: https://github.com/captn3m0/opml-gen

I believe that it might be against Telegram's ToS to scrape channels (Update: It doesn't seem to be). That said: I too really like the idea.

I had been thinking about writing a tool to scrape Telegram channels for another project, but found tg-cli with its Lua, Python bindings and weird command line interface to be quite messy (so I gave up for then).

Since this seems to be in decent demand, surely somebody has already written a tool for this?

I tried tg-cli and a few mtproto compatible libraries, but it was all a huge pain to get working properly before I gave up: https://github.com/captn3m0/opengram

The good APIs are the bot ones which won't work here.

telegram has this shitty feature where if anybody knows your public username they can add you to any group they want.

this is the source of the shitty spam on telegram right now. i was active in the crypto space and my username leaked. im now added to ton of spam and pump groups daily and i cannot block it.

also you dont need admin rights to the channel, but you have to work with https://core.telegram.org/mtproto to scrape channels. i did some side project with this https://github.com/zerobias/telegram-mtproto

How is this different to other messages where your phone number is the username? Last I checked this was possible with Whatsapp too.

your phone number is not your username on telegram. the username on telegram is like a handle on twitter, with the difference that on telegram you dont have to have a username. you can message anybody in your contacts without username. but if you chose to have a username on telegram that username is public, and searchable. and the worst part is that anybody can add your username to any channel.

so tltr: username on telegram is something you chose and not your phone number.

>your phone number is not your username on telegram

yes I know, haven't written that.

>and the worst part is that anybody can add your username to any channel

my point was that other messengers have public usernames too, mostly their phone number and suffer from that same problem.

> my point was that other messengers have public usernames too, mostly their phone number and suffer from that same problem.

the difference is important. the thing that is happening with telegram is the expected behavior, you are supposed to be able to add random online users with their username to groups and contact them, that is also the point for a username so you don't have to share you phone number.

with whatsapp your phone number is not supposed to be shared with random strangers, and sure if somebody has your private mobile phone number he can spam you but that is not supposed to happen, and i imagine that whatsapp blocks that behavior if you try to do in for multiple users. with telegram i saw groups with 50k members added this way because there is a api for this, but there is no api for whatsapp and i think they block you after x tries or reports. the problem is much bigger with telegram.

In Telegram: "Settings, Security, Groups" has options to allow everyone or just your contacts to add you to groups. Contacts alone is usually adequate, but you can add exceptions.

thank you. why would "everyone" be the default setting, i cannot imagine.

The reason might be to add & increase the number of users in any group.

yes, mtproto is such a horrible experience that almost every library is now abandoned.

The closest was this Show HN about a Channel-> Blog generator that worked on top of telegram: https://news.ycombinator.com/item?id=17617675

Nice. Most of you would know but some won't, some GitHub pages come with RSS feeds.[0]

[0]: https://github.com/DIYgod/RSSHub/releases.atom