Show HN: OPML list of Hacker News Users Personal Blogs | Hacker News Reader

Show HN: OPML list of Hacker News Users Personal Blogs

github.com

281 pointsoutcoldman99 comments

Now Is the Perfect Time for an RSS Renaissance (https://news.ycombinator.com/item?id=16907298)

I hope this list is constantly maintained and updated.

Additionally, do you have any recommendations for a good RSS reader that can properly handle this OPML file? I tried using Omea Reader, but it fails to fetch many of the feeds.

I was able to easily import the list into Inoreader.

> File exceeds maximum allowed number of subscriptions! Your file contains 659 out of 150 maximum allowed subscriptions for your plan.

Alright.. I'm gonna build my own RSS reader then.

Edit : A kind of "decentralized" social media platform where we post on our respective personal blogs, and we may follow each other by using RSS subscriptions. If HN is one such platform, here's the chronological timeline : :https://gist.github.com/altilunium/c2fcbe1e23aeb1cb9564f2593...

While generating this list, I found at least 80 sites with inaccessible RSS link : https://gist.github.com/altilunium/8319dae345f701a2a1efd0152...

freshrss is nice

Ah, I'm subscribed to their Pro plan, didn't know there is a limitation on this on free plan.

There is TTRSS already.

Recommend checking out https://github.com/miniflux/v2

n.b. I haven't tested importing 600+ feeds!

If you want a more casual approach then to right away import 600 feeds into your reader, check out the feed options at IndieBlog.page - it gives you daily random posts not only from the 600 URLs of that recent thread but also from a similar one back in April last year and a bunch of other indie sources...

https://indieblog.page/rss

Seems like my blog was not even included in the console output, but it has been part of the thread for a while. https://news.ycombinator.com/item?id=36596022

I know I don’t have enough karma, but it seems like the log output should have logged something for that. I also wonder if you’d consider account age as an alternative to karma for anti-spam, as I’m a bit of a hardcore lurker here and I highly doubt that I’m the only one.

(Sorry, I was wrong. See here for the author's reply:

https://news.ycombinator.com/item?id=36627385 )

You are definitely not the only one.

Everyone who missed the thread, please consider adding your blog here: https://indieblog.page/suggest

I've been playing around with Node-RED[1] for a while and thought I would recreate this using Node-RED (also being a big fan of Node-RED). The flow[2], i.e. code, is online to have a look at (editable but not deployable) and the feed[3] is cached and updated every hour or so.

It's only a small Heroku server so it might well be down or about to crash, I make no promises!

Thanks to the OP for the inspiration, I did take a lot of ideas from the original codebase :)

[1]=https://nodered.org

[2]=https://demo.openmindmap.org/omm/#flow/a8d90fc15ab6990b

[3]=https://blog.openmindmap.org/example/hacker-news-personal-bl...

>No feed: https://tommasoamici.com by tommasoamici ( 216 ) no feed found for https://tommasoamici.com

Uh, this made me realize I forgot to add the RSS feed link to the head tag when I recently migrated my blog to Astro, fixed now!

Hi Tommaso! Your board games collection is quite noticeable!! Bravo!

Same! Must fix...

Same haha. This was great to realize what i missed when rolling my own rss.

For others with similar issues I wrote up some documentation for publishing a feed that can serve as a checklist.

https://kevincox.ca/2022/05/06/rss-feed-best-practices/

Me too, added now!

There seem to be some malformed URLs here?

> <outline type="rss" title="ntietz" text="ntietz" xmlUrl="https:&#x2F;&#x2F;ntietz.com&#x2F;atom.xml" htmlUrl="https:&#x2F;&#x2F;ntietz.com&#x2F;atom.xml"/>

Unfortunately that is on u/ntietz, if you look at the https://ntietz.com, for some reason they encoded the href

``` <link rel="alternate" type="application/rss+xml" title="RSS" href="https://ntietz.com/atom.xml"> ```

Is that wrong or just unusual? IIRC entity escapes are allowed in attributes.

Hard to tell, web is weird :D I contacted the author and they have fixed it!

Looking at how many "https://url.com//feed.xml" there are in the list, I have a feeling that the scraping logic needs some work. Is it just concatenating "https://url.com/" and "/feed.xml"?

It's also not a good idea. On https://andinfinity.eu/ I have a header for that and it clearly indicates that the rss feed lives at index.html. Standard hugo thing I guess.

I was extracting URL from the alternate link in the HEAD of the blog website. The issue is that some people will do "//example.com/feed.rss", some "feed.rss" or "/feed.rss". So I built pretty stupid URL resolving logic, when I just should have used ResolveReference from go runtime. So the last list will fix that.

This is amazing! Thanks for the effort!

I missed the original thread, but would be happy when you include my blog as well :) RSS Feed (en): https://backendhance.com/en/blog/index.xml RSS Feed (de): https://backendhance.com/blog/index.xml

Best way to add it https://news.ycombinator.com/item?id=36627385

Thanks for the effort, this is a fantastic resource.

I installed 5 RSS reader apps in my Linux Mint machine to test the OPML. 4 of them failed to import the file: Thunderbird, Lifera, Quitrss and another one I forgot the name. Gnome RSS did import the file but it became slow and buggy.

I think the file is too long for a typical RSS feed app to process it. Also, there might be some formatting issues a couple apps threw a "incorrect format" error (Thunderbird being one of them). Now trying CPod and giving it time... I hope in 20 mins or so it can work with it. I wonder if web-based RSS readers will work better with the file.

I run freshrss and was able to import it there.

Do you get any errors while importing it?

I believe also this XML maybe not valid. Will regenerate it a few minutes.

I tried importing at feedbin.com, 524 of 598 imported successfully. The failures being a mixture of not found or connection errors and a couple of invalid URL errors

I just resubmitted the list, I believe around 500+ there is an invalid XML tag, which is fixed now. So all of them should be valid now. Forgot to do xml escape for the URLs

I had no issues with sfeed, you can get it from https://codemadness.org

Ugh, just need 20 more karma. Don't know anything about farming karma for hackernews though. Oh well, maybe I'll have to fork and allow users to specify their min karma to record the blog...

I will regenerate the list with just 2+ karma. Will let users to figure out what they like or not. 100 was just a random number I used without any idea what is good or bad.

I dunno, 80 seems like a pretty good cut off, let me just pull this ole ladder up after myself.... jk, Thanks! Love to be featured on the list, very cool idea. Feels like web rings are coming back

Is there an easy way to make a feed of feeds? So one RSS feed for all these?

That is good idea as well! That would require me to actually run the server (or github actions) to refresh the RSS feed, maybe something I would do later on.

Can you please ping me once this is up? We'd love to expose this feed in search results inside Kagi.

There are quite common and they're usually called planet <topic>, which might help to find useful resources or software

A github action updating a static file is, well, webscale. And cheap/free to host.

Just made something like that, though I had to split the feeds based on year, otherwise they'd get close to 100 MB in size: https://news.ycombinator.com/item?id=36643086

What's the difference between rss and opml? Just different standards?

OPML allows your to export/import the list of RSS/Atom feeds from your RSS reader. So this is a list of 500-600 blogs that people submitted at that HN discussion, that you can import in your RSS reader.

Apparently now RSS stands for "RSS Serves Structures" and OPML stands for "OPML Presents Mere Lists".

Or 'Really Social Sites' and 'Other People's Meaningful Lists'. ;)

OPML (when used in this context) is basically a list of URLs with a touch of metadata. It is the most common format for importing and exporting lists of feeds to move between feed readers.

So you generally don't subscribe to an OPML file, it just provides a list of feeds to subscribe to.

I made an alternative list of blogs with links to RSS feeds :)

[0] https://blogs.hn

https://hsnice16.medium.com/

Just a thanks for this. Wanted something like this from that list but didn't have time.

Loaded it into Feeder from fdroid (cheers to dev if here) and although it took a while, the result is much better than Twitter et al's output.

Might add my own blog to the list one of these days if people keep coming back to this as the reference.

Can we commit other people's blogs to this?

E.g. I am surprised to not see Julie Zhou's blog in this list--which, I believe, is one of the most influential blogs in the tech product space: https://www.juliezhuo.com/.

I just parsed everything that was published to https://news.ycombinator.com/item?id=36575081 so if they have not published their blog there, it is not in the list.

Let me send a PR to the project with this addition.

This is the first time to see useful OPML lists except export/import usage

That's really great! Thank you for that.

I saw some people asking for a good RSS reader. Personally I will use elfeed, and I also use elfeed-org so I could import the OPML file and export it into my existing RSS feeds.

I use freshrss with combination of Reeder for Mac/iOS.

My combination of choice too.

The fact that FreshRSS can fetch full article contents for truncated RSS feeds is precisely the thing you need in order to make RSS useful today. Not the biggest fan of the UI but this one feature makes the whole thing so fucking good.

OMFG somebody remembers OPML!

RSS is the only way to read news :D Reddit, HN - everything goes to my RSS ;)

Then why not use RSS, or Atom, or CSV, or HTML, for that matter?

That OPML "outline" has no tree structure at all, except for the single top level container item:

    <outline title="HN Personal Blogs" text="HN Personal Blogs">

What is the point of using OPML for a flat structure?

Outlines are all well and fine, but a list of links to blogs is hardly an outline.

And isn't JSON the simplest most ubiquitous accessible standard format for outlines and even arrays of links these days, if not just raw-dogging XML?

More thoughts and opinions and history of OPML, RSS, Dave Winer, More, Frontier, Manila, Radio UserLand, XML-RPC, SOAP, Xanadu, spreadsheets, CSV, JSON, XML, etc:

https://news.ycombinator.com/item?id=21170434

https://news.ycombinator.com/item?id=16224154

https://news.ycombinator.com/item?id=20780928

https://news.ycombinator.com/item?id=35674125

https://news.ycombinator.com/item?id=36544675

I think it's just that OPML has traditionally been used to import/export lists of RSS feeds, and therefore most readers are capable of importing an OPML file whereas other formats would not be as widely supported.

Why do RSS feeds need to be imported/exported as anything other than RSS feeds?

According to this thread, four out of five RSS apps on Linux Mint failed to import the OPML file, and the one that did was buggy:

https://news.ycombinator.com/item?id=36627431

Maybe I'm thick, but it bewilders me that someone wouldn't simply use RSS or Atom directly, instead of abusing OPML to mis-represent it (which is itself just as terribly designed as RSS, by the same person, Dave Winer -- don't even get me started).

Is this some kind of a masochistic corrupted data ingestion challenge from TikTok that I'm not aware of, akin to eating Tide Pods or washing your clothes in Campbell's Soup?

Does anyone even ever use OPML for outlines any more? Or is it like YAML's backflip, where its original name meant "Yet Another Markup Language" until somebody pointed out to their great surprise that it WASN'T actually a markup language, so they retroactively recursively renamed it "YAML Ain't Markup Language" on opposite day?

So does OPML no longer stand for "Outline Processor Markup Language" any more, but now stands for something else like "OPML Presents Mere Lists"?

What's wrong with exporting and importing RSS as RSS? Has its meaning also changed, flipping places with OPML, so it now means "RSS Serves Structures", so you use RSS for nested trees and OPML for flat lists now?

I suppose you could recursively encode outlines as nested RSS feeds inside the text content other RSS feeds as <![CDATA[ at each level, or since those can't be nested, exponentially doubling the number of escaped entities at each level.

I Wanna Be <![CDATA[ -- Sung to the tune of “I Wanna Be Sedated”, with apologies to The Ramones:

https://donhopkins.medium.com/i-wanna-be-cdata-3406e14d4f21

RSS traditionally kinda sucks for discovery purposes though. (Unless you count these HN threads as the discovery part)

I do wonder what's a good way of doing discovery in software, FOSS-style. Using open source client side and free APIs if needed.

What about subscribing to OPML files hosted on URLs ? That way one can follow OPML list "hosts" (like the one on OPs repository) and discover new RSS feeds. I'm quite surprised that this feature do not exists in RSS reader I tried.

Good. I'm tired of apps suggesting stuff to me.

This is the way. I just swapped as many news, social (reddit/etc) and the like over to newsblur via RSS and I'm VERY happy. This OPML will go a long way to filling out my categories :-) Thanks a bunch for your efforts!

I think my blog is not in the list even though I submitted it in the thread. https://aniddan.com/blog

Same, mine is http://wheybags.com/blog

Thanks! I saw the posts and have been making a move to newsblur recently, this will help a lot with content!

How do you update this list on a daily basis ?

Github actions is an option.

I am not sure if users of HN will be updating the original HN discussion in general, but yeah, GitHub actions can be an option for this.

For now, I just wanted to add those 600+ blogs to my RSS and filter them out slowly if I need to.

If possible, could you also add to description for anyone wants to add to your list, to go to original thread and submit ?

Currently it seems there're many people fetch from original thread, which makes it ne source of truth, less confusing for anyone to submit.

Yeah, the best way is to go back to the original post and submit comment there on top level. Make sure HN recognize that as a correct href and your blog has alternative link to rss/atom. And for now only users with 100+ karma are included.

I left the console.log there as well to show which blogs aren't recognized.

I am also open for PR. It was just a few hours project to get it going, but if people will find it useful, would be nice to get some tags from the blogs, so it would be possible to extract the list only for specific technology/topics.

> Make sure HN recognize that as a correct href and your blog has alternative link to rss/atom.

Any details on how to do this? My blog (https://blog.nawaz.org/) has an RSS feed here (https://blog.nawaz.org/feeds/all.atom.xml) but I don't know how to set it up the way you'd need it.

Put this line in html:

UPD: type="application/atom+xml" for Atom, type="application/rss+xml" for RSS

Didn't work for me and andinfinity.eu, but I'm in the other lists. Any ideas?

Added a note https://github.com/outcoldman/hackernews-personal-blogs/tree...

I checked yours, the issue is that you are escaping + in application/rss+xml, when it should be just application/rss+xml

I have updated the code and will re-generate. I am not sure if what you are doing is allowed or not.

I don't know Go's flavor of regex but is this correct parsing code: https://github.com/outcoldman/hackernews-personal-blogs/blob...

Are you taking care that if the <link> tag spans over multiple lines (linebreaks in between), you're still parsing it right?

You do seem to be resolving relative paths with:

  feedUrl := matches[1]
  if feedUrl[0] == '/' {
      feedUrl = url + feedUrl
  } else if !strings.HasPrefix(feedUrl, "http") {
      feedUrl = url + "/" + feedUrl
  }

So I think this is good. Just trying to see what could be breaking for the other users who are complaining their feed isn't listed.

I have fixed it for more people, should have used the built-in resolution func https://github.com/outcoldman/hackernews-personal-blogs/blob...

Ah, shouldn't have used a throwaway to post my blog. Oh well.

You can comment on that original post again with normal account, and I will regenerate.

Do sitemaps like `<link rel="sitemap" type="application/xml" title="Sitemap" href="/sitemap.xml">` work, too?

-> https://stackoverflow.com/questions/13327527/how-to-make-a-s...

Good stuff, thanks.

Needs a bit of lint work still, eg: current lines 12 & 13 are dupes

    <outline type="rss" title="bookofjoe" text="bookofjoe" type="rss" xmlUrl="https://www.bookofjoe.com/atom.xml" htmlUrl="https://www.bookofjoe.com/atom.xml"/>
    <outline type="rss" title="bookofjoe" text="bookofjoe" type="rss" xmlUrl="https://www.bookofjoe.com/atom.xml" htmlUrl="https://www.bookofjoe.com/atom.xml"/>

but still appreciated as that's a minor quibble easily polished away.

When do you plan re-run your crawler? HN users are updating that original HN discussion every minute as we speak! It'll be nice to get an updated list sometime today. Possible?

And yes, thanks for your work and sharing it!

I am fixing some issues, trying to get more blogs from the post, so do it a few times in a day.

And hope at some point to actually build something larger from that. I love RSS and this is the best way to discover new content. I actually already have found a few interested blog posts, links, information.

There seems to be many valid blogs that got skipped due not having the `rel="alternate"` link in the HTML head (mine included). I just added it, as I'm sure others will, so it'd be a good idea to update the list regularly

I just republished the OPML, now close to 1200 blogs!

Awesome! Great job on this

Oh, did it miss mine because I use rss.xml and not feed.xml?

https://blog.winricklabs.com/rss.xml

No, it's because you should include this link tag in the <head>

    <link rel="alternate" type="application/rss+xml" title="XXX" href="https://blog.winricklabs.com/rss.xml">

ohmygosh. Thanks, fixed.

Ohhhh! Thank you! Looks like I need to do this too for mine! (https://biggles.games)

Ah, no, I had the atom feed set up already, just insufficient karma on HN... Oh well!

I just gave you some.. who knows.

Yay, I'm listed!

HN is such a utopic place. I wonder when the inevitable enshitification will happen, and how?

Fear not, dang is Hodor.

It has its quirks on occasion and has changed a bit but has been around since 2007 and still going great.

Reply on news.ycombinator.com