Ask HN: Consolidated RSS feeds for independent blogs? | Hacker News Reader

Ask HN: Consolidated RSS feeds for independent blogs?

15 pointsPaulHoule11 comments

I've been experimenting with a smart RSS reader and I've been using Superfeedr to ingest content. Superfeedr monitors RSS feeds and calls a webhook when content arrives, I stuff these into an Amazon SQS and then spool them out at my leisure.

My system requires a fairly high volume (1000's of articles) to train a machine learning model so I have been focusing on high volume feeds like preprints from the arXiv, news from the Guardian, etc.

Now Superfeedr charges 10¢ per feed per month, my bill right now is $3 a month. It's reasonable that I can subscribe to about 100 feeds on Superfeedr but subscribing to 1000 feeds seems pricey to me.

There are a lot of independent blogs out there that publish an article every week or every six months. What they all have in common is that somebody has to poll an RSS file many many many times per article ingested. W/ superfeedr it makes for high costs but it is a hassle even if I built my own RSS ingest system.

One thing that would help would be consolidated RSS feeds that aggregate posts from a large number of blogs. Are there good ones out there? Are there other answers to problem of polling hundreds or thousands of independent blogs?

Back in the day these used to be called "Planets" and at one point there were lots of them with all sorts of content-specific niches and various levels of automation/moderation. (The best were opt-in/by-invite-only and extremely respectful to the bloggers the aggregated.)

https://en.wikipedia.org/wiki/Planet_(software)

You may want to be careful/cautious about what exactly from RSS feeds you train an ML model on. If you are planning to commercialize your ML model that may be a direct infringement on CC licenses many blogs provide their content as, especially to RSS and aggregators like "Planets". (This applies to using feed content you've already aggregated from Superfeedr as well.) Please use ML responsibly and ethically.

My system only creates classification and ranking models. I can't picture ever developing a generative model that can write new blog posts, it is much more likely I will pick up one somebody else trained.

By reasonable laws (US) the lack of copyright on a database extends to the trained model.

In the UK I can probably get sued for having a database entry that says I think somebody's blog sucks.

In the EU I guess I am supposed to know if some embezzler got out of jail and wants articles about his crime removed from the news everywhere so he can embezzle again. All I can say is I am glad I am not in the EU.

> By reasonable laws (US) the lack of copyright on a database extends to the trained model.

That's related to the point I'm trying to make: RSS feels like an "open database" with a lack of copyright, but it very much *isn't*. There's no lack of copyright on RSS posts. Most RSS posts explicitly mention things like Creative Commons licenses that apply to the copyrights on their content. There's an implied copyright on every blog by the very nature of blogging and its resemblance to traditional publishing (and traditional syndication). I'm deeply suggesting that with RSS you should start from the assumption that all posts are copyrighted by their author and that you don't have permissions to use the copyrighted contents. If an RSS item includes Creative Commons metadata you can start to use that, to the extent that the Creative Commons metadata matches your intended use cases.

I am not a lawyer, this is just friendly advice from an old RSS user and syndicator. RSS feeds being generally public doesn't mean that copyright concerns disappear. (This was one of the hard known lessons of some of the worst "Planets" that did things without author consent.)

When you've cooked it down to "article 7 uses the word ꜰᴏᴏᴅ 6 times" then it is a database.

If you want to go down that rabbit hole too far you start to wonder if those marketing emails people send you are copyrighted and if you shouldn't be able to store them in your email client and such.

That said there was an industrial petrochemist who got in trouble a long time ago because he had a huge file (multiple filing cabinets) of photocopied scientific paper that he used in his work.

The process of cooking it down that way is creating a derivative work and copyright still applies, it just gets more complicated. (Questions of "transformative" versus "fair use" versus license complications such as "no derivatives" and "share-alike" and much more.)

It's going take to years, if not decades, to sort out the copyright mess of machine learning. I take an extremely cautious approach to it myself, not because I think that the laws or the authorities will "crack down" on it any time soon, but because I ethically want to do the right thing (and took ethics oaths in the past, unlike so many in this industry; it's not fear of future enforcement, it's promises to myself that I'd try to do the right thing).

At the end of the day, it's about respect: I can't tell you what you are doing is illegal or not (and probably not even a lawyer can tell you for sure until far more court cases have tried or worst case you are at the end of your own personal trial on your specific use). I can point out that blog articles come from writers. Even writers not making money on their writing have implied copyrights. Many also have explicit copyright statements on their websites they blog from and sometimes also in the metadata of their RSS feeds (and you can't assume the lack of one implies the lack of the other). Respect the writers as best you can. Don't just treat them as faceless metrics to be compiled into your models. That's your responsibility in training an ML model. I would suggest doing it cautiously, but that's my sense of ethics talking, I cannot tell you what your ethics should be here in this moment, only suggest what I find ethical or not and that I'd probably be a lot more cautious.

Planets were cool. There is still a planet at ubuntuusers.de: https://planet.ubuntuusers.de/

There are still a bunch https://planet.postgresql.org/ https://planet.haskell.org/ https://planet.lisp.org/ https://planet.scheme.org/ https://planet.emacslife.com/ https://planetgolang.dev/ and many more

Is there a directory somewhere?

Why don't you just get a list of feeds and scrape them? It doesn't sound like you need real-time updates for your use case so polling shouldn't be very expensive. You can just poll weekly or less. Polling a thousand blogs shouldn't take long at all.

A good feedreader wouldn't poll again when it receives a 304 response.

Lambda, cloudflare workers (probably cheaper) or GitHub actions + a repo

Reply on news.ycombinator.com