The Rise and Contentious Fork of RSS
twobithistory.org
twobithistory.org
The title is misleading.
People who captured the feedreader market, killed the competitors then dropped it did a lot of damage. Maybe deliberately. LESSON: Never trust a big company that relies on advertising.
RSS is a way to escape the nonsense that has come with those, who have no idea what the web can be, but have flooded in and drowned so much in their ignorance and lack of vision.
All: let's focus on something other than the title now—it's a pretty interesting piece for anyone who doesn't know the history (as I didn't).
Still...
Google Reader did more to destroy my perception of Google's reputation than everything else evil they've ever done. Conspiracy to suppress wages for tech workers, James Damore, the Snowden revelations - all of those bugged me, but not like Reader.
They killed Reader, I suspect to support Plus, just as the walled gardens of the social networks turned into iron curtains, destroying the only viable alternative decentralized content distribution mechanism. And now we get to live in that world.
Reader's shutdown, after monopolizing neAly RSS tech and in order to push their own flavor of the year (Plus), was a classic tech evil
The other evil that Google is perpetrating is AMP. Reader followed an "embrace and extinguish." AMP is a full on "embrace, extend, and extinguish."
That's the closest I've found to the cleanliness and speed of Google Reader, with additional features
Now, you might argue that "But Google Reader was the de facto standard and mainstream audiences didn't know to just use something else."
However, unless they were reading RSS in the background through a portal like Yahoo!, mainstream audiences weren't using RSS long before Google Reader was killed (if ever).
Explicit use of RSS through RSS readers was always primarily the province of journalists, analysts, researchers, and various other folks who wanted relatively unfiltered from-the-source on specific topics.
It also didn't help that individual blog content has tended to migrate to publishing platforms in general. I still write a lot but publish on my "personal" blog relatively seldom these days because visibility is so much higher elsewhere.
Google Reader nailed the social experience around RSS, and there definitely was a high point where RSS feeds were used by a lot of people on Google Reader that didn't understand and didn't need to understand RSS feeds to take advantage of them. It was just that weird news app "next" to Gmail on their browser tabs.
At least that was my anecdotal observation. A lot of college friends that I tried to convince to use RSS for ages before suddenly used RSS because it was already lit up on their Gmail account and they were seeing the stuff I was sharing at the very least.
Let me rephrase that. Never trust people more powerful than you to consider your situation.
That is basically also a core principle of the distributed building blocks of the internet. It's just that to the masses and to the power hungry these options are not attractive.
If it matters, it's probably in an RSS feed. It's social media blithering that's left out. And ads.
I am not particularly bothered by the lack of RSS on social platforms.
You used to be able to do that. The problem (from fb's point of view) as the sibling comment points out is that you'd never visit facebook.com unless you wanted to actually wanted to interact with the content.
Apparently Mastodon generates an Atom RSS feed for every user, https://news.ycombinator.com/item?id=16755553
Link: https://www.bloomberg.com/view/rss/contributors/matt-levine....
Oh, and there's also this https://www.bloomberg.com/politics/feeds/site.xml
http://www.smashcompany.com/technology/rss-has-been-damaged-...
I'm speaking from my own experience running Feedity - https://feedity.com, a growing startup that helps with custom feeds for unstructured sources like webpages.
(RSS is, like, objectively the best way to follow multiple webcomics. I don't understand why it's not more popular.)
1. You miss out on the aesthetics each comic's website brings to the table
2. Sometimes you miss things like "hidden panels" or blog posts you're interested in.
3. No advertisement money going towards the artists (Not a big deal if you support them by buying merch and the comics don't need a lot of bandwidth)
I would really love to develop a reader that focused on webcomics, having both a dynamically curated section as well as all of your subscribed comics. Newspaper-like layout. Customized borders around each comic to add back some of the flair each comic's website offers. Quick links to the artist's blog/website/comic permalink. Text/image-only advertisements interleaved between comics, with 100% revenue going back to artists. Even merch advertisements which occasionally appear under an artist's comic, offering links to their storefronts.
Basically an open source Funny Papers for the web, which would offer content discovery for artists through its curated/related sections, as well as revenue from advertisements. It would update daily and could be used to follow both new and old comics serially.
I just need someone to help me build it because I'm juggling too many things to devote all of my attention to it.
The way I use my RSS reader is that I scroll through the list of posts, and if it's something I wish to check back on later, I press the shortcut to send it to instapaper, and if it's something I wish to check now, I press the shortcut to open it in a new tab, and then I go on. When I'm done, all posts have been marked as read, and then I have a few tabs open that I immediately check out, and a few links in instapaper that I check out later. I never actually read an article or a comic or a video in my reader. It's terrible interface, I find. I much rather check out the actual website. I do the exact same for instapaper; open in a new tab and archive the link.
With the exception of web apps / extensions, it's my understanding that RSS feeds were originally meant to be consumed by a program specifically optimized for handling feeds, instead of a bloated, slow, unsafe web browser.
In the end, your RSS flow could mostly be replaced by a set of live bookmarks in your bookmark toolbar, then you wouldn't even have to leave the browser. Plus you would get easy multi-platform synchronization of your feeds. A hell of a lot easier than running an RSS reader on a private server like a lot of us.
Also, my RSS reader is just a web app, not a native application, so when I get through my list of new posts in my feeds, I'm already in the browser.
The core feature of RSS is that it takes content and moves it from a site the publisher controls and into one the reader controls. This is immensely valuable to readers! It's perhaps less valuable to publishers, however.
Many publishers invest significantly in adding value to and gaining value from their websites. Advertisements, links to other sites, merchandise sales, etc. Almost all of those vanish when you feed only the core content to readers through RSS.
With all that in mind, publishers have ample reason to not invest in RSS. The economics of it are not kind to them.
The only people left using RSS will be the classic web of independent readers and writers, who are still free to donate directly to each other with only a payment processor as middleparty.
At the same time, it's worth considering that those publishers are often much worse off when they escape the cat's malicious paws. More than one newspaper has learned this to their considerable sorrow.
The comment above where someone said "RSS isn't dead, I have a business scraping pages to make RSS feeds when they didn't have one already"... yeah, exactly.
Only a couple are available yet annoying, like the forum at eham.net, which offers a feed that essentially gets you unsorted forum replies mixed in with new topics. So every day's updates are full of "Re:" entries. Anyway that's been the only speed bump so far. I was surprised.
Also, since I monitor niche subreddits on Reddit, their generalized feed availability means I can automatically get amazingly wide access to a variety of niche topics (e.g. shorthand writing).
Part of my experiment was aimed at figuring out whether RSS is useful to me. It really is, though careful culling of unwanted or unneeded feeds is important. It's kind of like maintaining a plain-HTML links page. Still valuable in its simplicity, but also requiring ongoing refinements to continue to be useful.
I honestly have never used a feed aggregator, I used Slashdot, then Reddit, and now a mix of Reddit and HN.
Having a filter of some sort is really useful.
I want to read every XKCD comic, so I subscribe to its RSS feed. I like to read a little tech news when I'm bored, so I go to Hacker News.
I 'consume' other people's thoughts and opinions via HN, after 'consuming' the story via my feed.
Different purposes.
So, more Americans use RSS every month than use Twitter.
For podcasts you need one mp3, usually quite compressed. For video you usually need more than one source, adaptable stream quality and good seeking (because most people do not watch videos end to end but only view parts of it).
This is probably not insurmountable but I guess we need better tools for creators to abstract all of this.
A 'podcatcher' (ie whatever app downloads it) is downloading the 'enclosure' url (which can point to any type of media) of the latest RSS 'item'.
Yes! Check out the deck and video from Edison Research's "The Podcast Consumer 2018" at http://www.edisonresearch.com/podcast-consumer-2018/.
https://github.com/feediron/ttrss_plugin-feediron
On occasion I have needed to use a site parser like rss-bridge but honestly it's so rare that a site doesn't have rss (albeit often hidden from view). Just a hint for people searching for rss feeds the most common I find are:
/rss
/rss.xml
/feed
/feed.xml
/index.php/feed
/index?format=xml
Today, nearly all Web feeds are available as Atom. Hackers, learn from this. Specs matter.
https://web.archive.org/web/2004/http://diveintomark.org/arc...
Atom exists precisely because RSS is a clusterfuck that could not be salvaged.
> The article downplays the significance of Atom.
Does it? Atom Syndication Format seems like a vestigial feed format, so I'm interested in cases where it's used and RSS isn't also available.
@dang Congratulations, you just invented OStatus. :-)
OStatus is a combination of Atom+ActivityStreams+Salmon+WebFinger+WebSub for exactly this purpose. It was created in 2009 and was even partially used by Google Buzz. I could post something on identi.ca and see it pop up in my timeline on Google Buzz. But as you indicate regarding RSS, Google didn't know how to monetize it, and they soon killed Google Buzz in favor of the closed Google+, years before they killed Google Reader.
When Mastodon arrived in 2016 it used OStatus as its primary protocol until the new ActivityPub was recommended by the W3C as a replacement. OStatus is still supported by Mastodon as a secondary protocol, and part of the Fediverse still runs on pure OStatus.
Diaspora was inspired by OStatus, but removes the WebSub part in favor of going full Salmon, and does a few of the details differently.
"WordPress powers 31% of the internet.", according to wordpress.com and Wordpress has RSS built in and ON by default.
that's a really general statement but I'm going to bite. Tech need not be up front and in everybody's face. I think HTTP is the same. It seems reasonable that most people don't know or care what http is or what it's doing at the beginning of urls. Or why it sometimes has an 's' on the end. And I'm even more confident saying that 'most people' don't know about how amazing TCP over IP is.
I'd like RSS/feeds of strongly typed content to be more invisible and prevalent. We're living in the HTML dark ages here.
Should probably need to know how RSS actually works.
https://github.com/damng/hackernews-rss-with-inlined-content
This generates an RSS feed with the contents of the articles inlined and available at https://damng.github.io/hackernews-rss-with-inlined-content/...
By-passes most soft paywalls. No Javascript. No Tracking. No 'social' media buttons. No modal dialogs. Just text you can read in a terminal based rss reader like canto.
This is why I absolutely LOVED FriendFeed. Not only could you follow arbitrary RSS feeds and FriendFeed users, but you could group feeds into "imaginary users". This was great because you could follow anyone you wanted, even if they didn't use FriendFeed themselves.
So what happened to FriendFeed? Acquired by Facebook and then killed.
It's primarily microblogging-oriented, but you can subscribe to any RSS feed too. And users (including RSS feeds) can be grouped, so you could have one group "tech feeds", one group "cat accounts" etc.
With the shift in usage to the edge, RSS found itself in the company of adhoc webpages that disseminated the same content but supported scripted adtech to enable just-in-time display advertising -- advertising which, over time, became a significant source of publisher revenue. RSS was often surfaced in a different user-agent and unable to clearly accommodate scripted ads, so first-party RSS feeds were practically giving away the content for free. With time, this greatly contributed to the waning appetite of commercial publishers for RSS, despite it remaining popular in circles were the content was meant to be spread wide at no cost. Although many news organizations still offer RSS feeds, they're not really marketed prominently, and remain a loss leader to make savvy readers happy.
Since alternate business models never achieved the same uptake, efforts like Google's AMP (and its offshoots like webpackage [1]) are a practical take on the syndication concept again, where the usage of adtech is assumed from the start, and the surfacer of the content becomes decoupled from its author.
This has happened in at least one media segment: podcasting. Ads are simply inserted in the medium proper, or the feed gets paywalled, or money is generated while producing the content (live shows).
It is entirely possible to reproduce this achievement in other media sectors, it just needs to be done wisely:
1. a catchy name ("rss" or "atom" are too geeky, "feeds" or "syndication" are too generic, "newscasting" is another thing).
2. leave clients and formats independent, so the immediate needs of users prevail over broadcasters' ones. Broadcasters are good at telling people what to think, but terrible at telling them how to use or consume anything. It also removes the petty fighting like "we won't rely on a platform built by $competitor".
3. Leave ad-management and general money-making schemes up to publishers. They can license feeds to specific clients only, they can do paywalls, micropayments, embedded ads, whatever. The client should assume nothing, simply move content from A to B on a schedule. All this can easily be implemented on HTTP codes, they are all there already.
4. The real point of feeds is exactly this: move stuff from A to B on a schedule. Metadata is almost irrelevant. Content display is a side-show. What podcasting delivers is control: it gets the content a user wants, as soon as it's available, and makes it ready to be consumed when the user needs it, in the way s/he prefers. This concept is what should move the effort, not dreams of semantic sheeps.
5. Because it optimizes pre-fetching, the format shines where on-demand fetching is time or money-expensive. Niches where it would (or does already) work well, ae things like auto-downloading TV episodes, delivering big images (something for visual artists to cut off Pixiv and friends... ?), delivering big binary updates and so on. With the rise of single-page-applications, one could even devise a format that delivers an entire website, ready to be consumed instantaneously without requiring lots of mobile data usage. Bootstrapping it in "stingy" markets (Southern EMEA etc) could help.
6. Good directories for clients to use. Podcasting really took fire when good directories emerged.
(on a sidenote: AMP is evil and publishers are shooting themselves in the foot. It seriosly risks to become an "iTunes" moment, where an entire industry relinquishes control of their market because they don't understand long-term consequences.)
> They can license feeds to specific clients only
this is a bad idea.
And probably also: Are you aware that the request to work in email somehow is a joke with about 90% certainty?
Just a quibble about terminology. Those handful of corporations do not control the web; they merely dominate it. It's an important distinction. Actual control is censorship, which is an important issue these days. The fact that a lot of people are using systems associated with a few companies is in the end relatively unimportant.
RSS is quite important simply because it is used to discover individual articles from lots of different sources. People who post things with RSS feeds are not raging against the dark but instead are routinely maintaining the light.
That didn't take long to find. Why use that as an example?
Once I start finding sites that don’t support RSS, then I worry.
But if you care about what content is available, if you care about how people who create work you enjoy are able to make a living about it, if you care about things like who's the president of the US. Then it might behoove you to care about the fact that between them, Google, Twitter and Facebook control the majority of the referral traffic for the most popular 100 websites. A future where RSS prevailed over social media as a way of curating content for most people is a different world than the one we live in, and if you care about the world you live in, you might care about why RSS didn't prevail, even if it's still usable for the minority who prefer it.
Reread my post. There is a difference between caring about whether the majority of users are using RSS and caring about whether publishers are creating RSS feeds.
I specifically said “Once I start finding sites that don’t support RSS then I worry.”
For instance what the exact format is does not matter. Supporting RSS, Atom and a few similar formats from a reader perspective is just no problem compared to parsing bad HTML, dealing with JS Hell, etc.
The real issue is a lack of innovation on the part of feed readers, particularly that we still see the "here is a list of all your feeds" viewed separately and no effort to make a workflow where you can make it so your time is not wasted by articles from Tedium and sensationalistic sites like Fox News and the New York Times, etc.
see
http://ontology2.com/essays/HackerNewsForHackers/ http://ontology2.com/essays/ClassifyingHackerNewsArticles/
the machine learning technology to do this is ten years old and if the kids out there weren't so interested in word embeddings, functional programming and other fads, they could bring this to the people.
Content discovery for the everyday internet user consists of either Facebook, Google, or Twitter. And none of them are opaque about their SERPS / timelines.
RSS is/was nice to be able to stay up-to-date with infrequently updating blogs, and it's honestly a shame that it's no longer used as much as it used to.
One thing that could possibly re-ignite the use of RSS feeds is a button in the adress bar (I recall Safari having this option) to indicate that the site has an RSS feed and taking you to it when you click on it. Maybe look in a couple of predetermined places to find this feed (feed.xml and rss.xml are almost universally the common feed names I've seen used on the sites that I frequent, with the exception of reddit, which simply lets you append .rss to any URL).
I still use it every day, its a vital part of my controlling what info I see every day process...
I read a post about the contentious history of RSS several years ago, similar to this one, and that bit really jumped out at me. I'll see if I can find it again.
The demand is how to get the content of what you want and when you want in the quickest and easiest way.
Social networks, Tweets, FB, RSS subscriber, flipboard etc. all attempted despite the Ad revenue must be taken care along the way, yet none has really met the demand.
Do you get news from your friends, or subscribe people, or visiting sites? To me it is as much as mundane as 20 years ago when internet started. The problem has been resolved except many have made tons of money while tackling it. The fundamental problem remains unresolved and still waiting for answers.
When RSS concept started we simply do not have always-on tech, ads-tech, aggregator's tech and software itself was a business. Now that we have the above TheBig simply prefer control their consumers better. RSS-web-client like Google reader are simply ancestor of modern aggregator, a way to control users while leaving a bit of freedom, aggregator's suppress this last bit of freedom, that's for me the real reason behind RSS/Atom decline.
Sorry for my poor English.
edit: There is also the news that NetNewsWire is back to its original developer and will be released later this year: http://ranchero.com/netnewswire/
> Apple's stance on privacy and security applies only if you don't back up your data to iCloud ... In the first half of 2015, police agencies around the globe asked to explore 4,472 Apple customer accounts, according to the company. Apple disclosed data to police on 1,886 them, of which 1,407 were provided to law enforcement in the United States. In these instances, Apple gave investigators contents of customer iCloud, iTunes and Game Center accounts. Apple didn't always turn over the information. It objected 401 times in all.
"Exploration" of iCloud accounts is remote, e.g. ongoing surveillance in search of data to justify a warrant. It can be done by law enforcement from multiple countries globally, while searching of a house is performed by local law enforcement operating under local law.
Here is an incomplete list of various problems you'll have if you want to write a feed reader http://inessential.com/2013/03/18/brians_stupid_feed_tricks
It requires a lot of work to make a feed reader that supports all possible feed generators from all possible sites. And it's more related to multiple buggy producers than to multiple feed formats (for example repeating item IDs, different item IDs for the same items, missing IDs -- no matter what format you use you get same issues).
Another complexity is feed updating itself. Silo hosts content and knows what was updated. Feed readers don't. So they need to poll and update a lot of content even if they have just a few users (but with thousand feeds each). There is a PubSubHubbub, but again, lots of providers and implementations could be buggy (for most feeds I just fetch them directly when receiving a push instead of use push data itself since it's more reliable).
So the open nature of web syndication itself dampens its usage by making developing and running feed reader a costly task. It's really simpler (and maybe more profitable if you're lucky) to write a silo.
And there are more problems that are not related to feed format at all: relative complexity of subscribing, not everybody need syndication (many people prefer occasional visiting of the site instead of reading each article), new forms of "syndication" (related videos, algorithmic feeds), less blogs (maybe it's good -- let people post cat pics in social media), less popularity (not a part of big sites, browsers), less education (social media and messaging apps is the only Internet some people know) and so on.
But there are people who want to follow every article from interesting sources in software optimized for this task. And feed readers are the best thing here. For example, some people use my feed reader for Facebook/Twitter feeds only (no RSS feeds at all) just because it's more convenient to read them here.
In the end, I think, process of reading subscriptions is more important than underlying format. If RSS will die (quite unlikely) another methods of getting articles will appear.
While I was building it I did notice it had become harder to find a poscasts RSS feed but they definitely all still used it. I suspect it is used more as a business to business/platform to platform transport these days.
Certainly beats building thirty API integrations for people who want to roll their own syndication for their podcast.
Or use the API: https://www.listennotes.com/api/
How is the audience feedback so far?
June, 2018: https://www.producthunt.com/posts/listen-notes-3-0
Dec, 2017: https://news.ycombinator.com/item?id=15825900
But one comment that I heard again and again is: It's meaningless to build a podcast search engine, because Google and Apple can easily build something similar :)
As the episodes are syndicated it seems reasonable to "rebroadcast" the episode on your site, but the terms and conditions for many podcasts (example 99 Percent Invisible) seem to suggest that is breaching their usage rights.
Have you looked into this at all and did you find similar issues?
Also... that website proudly displays an RSS feed button on every page.
Recently had a client that needed me to read their RSS feed into email templates and update their template after they switched their email service. First time working with RSS. 10/10, would do so again.
Already a standard! https://jsonfeed.org/
- It is really just RSS/Atom wrapped in JSON. Sure it changes a few names, makes some elements mandatory and other elements optional but overall it does not attempt to fix core problems (consistent item IDs to avoid reruns, embedding messy HTML with relative links...).
- Besides a dozen of JSON feeds mentioned in the documentation or in the Github repo, there are very few feeds published in the wild.
- Its authors seem to have kind of given up already, a lot of open issues and PRs but not much happening.
Don't get me wrong, I'd like a successor to come and fix the issues that RSS and Atom have but this successor would need much more traction than what JSON feed has.
. $x is dead
. the rise and fall of $x
. new battery breakthrough claims to $x
. Ask HN: $question_where_the_answer_is_my_own_business
. <Google|Apple> <unveils|retires> create_rand_product_name($technology)
. Study reveals random_food() <increases|decreases> risk of random_quality()
. <any string of english words that contains "Machine Learning" or "AI">
. A random_quality() RayTracer in random_tech()
. ??? suggestions welcome
This would then force hn to have some sort of filtering/moderation.
Randy's law 247: The easiest way to fix a non-trivial system is to abuse it.
Ask HN: $question_where_the_answer_is_my_own_business is my personal favourite.
Normally a rise and fall, etc, is only a title looking for eyeballs - what's most odd about this otherwise well researched article is that the last two paragraphs actually proclaim RSS's death, when this is clearly not the case. It's not as prominent without a flagship corporate app like google reader, as noted by the author, but it's nearly everywhere and there are a large number of very usable readers on mobile.
It is a good argument against corporate walled gardens, however.
While we are at it, we should also bring back Usenet/nntp (though obviously I have nothing against news aggregators with discussion boards such as hn). But long term, only a new distributed p2p web will be able to escape the ever-increasing shilling and censorship. You can start this process right now and be prepared for the future by separating content strictly from web apps and avoiding JavaScript-heavy approaches for content.
That's funny, I see the same legislation as the death of all unauthorized RSS aggregation. The "link tax" provision isn't about links, it's about excerpts of content.
That would imply people will go back to installing things, hunting for feeds, copypasting urls... As much as I like desktop tech, that boat has likely sailed, at least for the mass-consumer market. You might maybe get away with a mobile app, but you'd still need help from browsers to "subscribe" to feeds - something that was pretty awful the first time it was tried, and has been basically abandoned or removed since.
On the desktop it would be harder though.
Nope. We could have a simple list of feeds, with a simple 'add' button next to the site name, even embedded in the app itself, as preferred/most read feeds list. Since this list contains no excerpts, and actual excerpts and feeds would be coming from publisher themselves, it shouldn't be an issue.
Of course, then we have questions like order of this list, how many and which feeds it lists and all, but it offers enough convenience that if made avoidable, we shouldn't have a problem with it.
Or since IANAL, this _may not_ be how the links and excerpts are considered, and I'm wrong, in which case, I'd like to know so I can correct myself.