Tell HN: I curate HN stories which didn't reach the front page

Edit: ouch! - it was a moderator who added "Show HN" and that was totally our mistake. I'm so sorry for scolding you for something you didn't do!

Newsletters can't be Show HNs. Please read the rules—they mention this explicitly: https://news.ycombinator.com/showhn.html. I've replaced "Show HN" with "Tell HN" above.

Also, if you're getting these stories from HN then I would ask you to link to the HN submissions.

We totally support the goal of bringing attention to great stories that haven't gotten significant attention yet. Other people have worked on this problem (e.g. https://news.ycombinator.com/item?id=23392049), and in fact we spend a good deal of time doing the same thing (https://news.ycombinator.com/item?id=26998308). But it's not good for community to sever the links to the source where you found them. In fact, at first glance it felt exploitative to me—a bit like kicking the ladder out from under you—but on reflection I'm sure that's not what you intended.

If you include the HN sources the way that polote has done here: https://hnblogs.substack.com/p/hn-blogs-24121, that would suffice.

p.s. It looks great and I'm going to subscribe!

this page should probably be linked to on https://news.ycombinator.com/newsguidelines.html, I had no idea there were specific rules for the 'Shown HN' prefix

It's in the FAQ: https://news.ycombinator.com/newsfaq.html, as well as at the top of https://news.ycombinator.com/show.

Btw, it's been on our list for a good 7 years to ask people to confirm that they've read the rules before posting a Show HN, but I always hold back because it would break a certain precedent. HN doesn't tend to use nag screens and so on.

at first glance it felt exploitative to me

Exploitative? Of who? The submitters?

I only ask because as both a submitter and content creator I have zero problem with anyone finding items through /newest and sharing them wherever they like, credit back to HN or not, and am unsure who benefits from dissuading this.

No, exploitative of this community. If you find things through HN then you owe that to the community and should acknowledge it—I don't mean in one-off cases, of course, but in cases where someone is building an entire project (such as a website or a newsletter) on it. In those cases, credit for that provenance should be given in the form of links. That's the way to balance giving and taking, and it allows readers to retrace the steps and encounter the community for themselves, which they may find interesting. FWIW I have the same response when people build job websites off of Who Is Hiring threads.

> but in cases where someone is building an entire project (such as a website or a newsletter) on it.

Oh, this is an interesting qualifier. The newsletter is not based solely on HN. There are other sources as well. How much is from HN? I'm not actually sure myself, but I'm gonna attribute some HN links (where applicable) and we'll see.

I'm not sure that what I said above was particularly clear, so I feel like I should rephrase it. If you already understand this and are including links to the places where you find the articles, that's already the solution and I don't mean to pile on! But maybe others who see this will appreciate a clearer explanation.

Whatever sources you're systematically drawing on, each deserves acknowledging. By "systematically" I just mean that when people randomly pass around one-off links to random articles, there's no need to be particularly scrupulous about provenance, but when they're building projects on other platforms, that's different, and then there is such a need. Otherwise the balance between giving and taking is out of whack.

The "taking", in this case, is using HN (and other aggregators) as a source of links to interesting stories; this needs to be balanced by some "giving", and on the web, the well-established way to do that is to link back to the place you found it. That way, good things can flow in both directions: your readers get interesting things to read, but they also get to know about a community they might like, and the community has a chance to gain new readers who might enjoy it.

If you don't do that, then the claim you're implicitly making is that you scoured the entire web looking for those articles, rather than drawing them from other aggregators that already did that. That woudl be false and misleading—it's "taking" too much, in the above sense. What you're actually doing, which is looking at existing aggregators to find good content that got overlooked there, is (a) genuine work and (b) a wonderful contribution—and that's the work and contribution that you deserve and should take credit for. I don't mean to criticize that or minimize it! The existence of many good articles that get overlooked shows how much it's needed. It's simply that each contribution needs its proper acknowledgment.

Some of those featured stories did reach the front page.

Perhaps it was curated prematurely for the newsletter?

I don't set out to only curate stories which don't reach the front page. That's just a side effect, not a criteria.

Also, sometimes a story reaches the front page after it appears in my newsletter because it got re-submitted. Correlation? Don't know.

Clickable links;

[1] https://bengtan.com/interesting-things

[2] https://bengtan.com/newsletter/sample (redirects to the latest public edition)

Cool collection of articles. One thing I’d like is a link to the HN discussion, for the articles taken from here.

Hmmm ... I thought a lot about this ie. whether to include the HN link. Problem is ... when I write the newsletter, most HN links are empty or only have one or two comments.

Later on, the story reaches the front page on HN but it's a new HN link. So if I include the 'original' HN link, it's not useful to the reader, and may even mislead them into thinking there was insignificant HN discussion.

For example, I linked this story:

[Web Applications from the Future: A Database in the Browser](https://stopa.io/post/279)

in early May but it only recently garnered significant discussion a month later in early June (https://news.ycombinator.com/item?id=27424496).

So I'm not entirely sure how to resolve this.

You could use the “past” link that leads to Algolia search for the story.

In the case of your example https://hn.algolia.com/?query=Web%20Applications%20from%20th...

But not sure if that’s a good idea or not. Just suggesting it as one alternative to consider

I'm gonna think on how to solve this conundrum in a way that satisfies everyone. This is one data point I'll think about. Thanks for bringing it up.

But then your traffic would drive that HN discussion and then comments would appear.

Interesting. I didn't consider this.

I'm gonna think on how to solve this conundrum in a way that satisfies everyone. This is one data point I'll think about. Thanks for bringing it up.

I hope you think on this. On principle I think it’s wrong, but personally I care far more about any sort of discussion in general. I want some sort of link to a [higher quality] discussion or potential of one. I enjoy HN comments and the community. Trusted above others like Ars comments.

These links are useful even if they don't have a discussion:

1. It's interesting to see that a story has been submitted many times, for instance. This can be seen by clicking on the domain name in brackets after the story

E.g., a few days ago I posted a link to Heidi Howard & Ittai Abraham's 'Raft does not Guarantee Liveness in the face of Network Faults'

It was the fourth time this 6-month-old link had been posted, each time attracting upvotes but not enough to get on the front page. (I think it usually takes about five upvotes for a story still on the first page of 'new').

https://news.ycombinator.com/from?site=decentralizedthoughts...

2. Given that your newsletter mostly contains links that are less than a week old, the discussion link will still be live. You seem to think discussion is most likely to happen on a resubmitted link: I'm guessing your newsletter's readership is unlikely to be large enough to get the regular kind of front-page HN discussion going for many of the stories in your newsletter, but the opportunity to comment can still have value even if only a couple of comments are posted.

3. If a story is well-written, I generally like to glance at other things the author has written, and looking at stories that have been upvoted here on HN is often a good place to start.

I’ll second this. Looks useful, but I really want links to the HN discussion along with the article (and maybe to the Reddit or other sources in those cases)

For what it's worth, I made a Firefox extension that will light up whenever the current page has been posted to Hacker News. I find that it helps me in situations like this, and also adds a lot of depth to regular Internet browsing.

It uses a giant Bloom filter with every article ever submitted to preserve user privacy.

If you normally enjoy being linked back to HN discussion, maybe you will like it!

https://github.com/jstrieb/hackernews-button

Thanks, I’ll check it out!

It's worth noting that my extension is far from perfect – it turns out that determining whether a specific page has been submitted to Hacker News is far from a trivial problem to solve. In general, this is because multiple URLs can map to the same page.

Direct string comparison of the current URL to previously submitted ones doesn't work because there are many ways for two identical web pages to have different URLs. For example, the URL fragments can differ (the part after the "#" that may or may not be present). Also there can be tracking parameters (often—but not necessarily—prefixed with "utm_"), which don't change anything about the page. But the URL parameters can't be entirely disregarded because sometimes sites, forums in particular, rely on them – consider pages that use an "?id=..." parameter for different pages. Thus some parameters should be removed, but some shouldn't. The same website having different domains (or domains that change over time) further complicates the situation.

My solution was to "canonicalize" URLs by transforming them into a simplified form using some pretty rough heuristics for common sources of noise. The Python code to do that is here: https://github.com/jstrieb/hackernews-button/blob/master/can...

All of this to say that even though I've used my extension for months and have been quite happy, there will inevitably be false negatives.

A solution designed to be used by search enginges is the canonical link element[0] present in the html of the page. I'm not sure how this would work for a browser extension, as you would have to crawl every submitted site once and save the canonical version.

[0] https://en.wikipedia.org/wiki/Canonical_link_element

Why, newsletters?

I never sign up of extra emails... But I do occasionally add stuff to my RSS reader..

Which is what I read when HN frontpage has nothing interesting.

Most of the time, because they can't capture leads with a freely accessible rss feed.

> Why, newsletters?

Amongst other things, I'd like to build up a mailing list. But I realise newsletters don't work for everyone, which is why I also post to a freely available web archive.

> Which is what I read when HN frontpage has nothing interesting.

Well, the next time you have nothing interesting to read, feel free to peruse my newsletter archives. :)

> I'd like to build up a mailing list

Just FYI that's why I (and I assume the OP) don't sub to newsletters.

It is annoying, but different sides and people are merging newsletters and rss. You’ll still get the mailing lists’ new normally unwanted different emails, but at least it will be contained so easier to squash and won’t hit your actual email.

> Just FYI that's why I (and I assume the OP)

Exactly, whenever I see "newsletter" I think spam.

Because clearly your content isn't good enough that people will come back on their own.

I do something similar, daily, with Plurrrr: https://plurrrr.com/ . It comes with both a JSON feed and RSS feed.

You don’t link to any discussion either? Already said this. Both personally and on principle, it doesn’t seem right.

I do link to discussions if there are any and if they add to the link itself, for example: https://plurrrr.com/archive/2020/05/13.html

This is great ! In fact, this got me thinking about what a simple ML model for automating HN's 2nd-look could look like. One could assume that the log odds of user_a' upvoting an article, after visiting it off the 'new' page, is a linear function of

* the article's quality (denoted by x) and

* user_a's click happiness.

P(upvote_a | seen_a) = exp(a + λₐ x) / (1 + exp(a + λₐ x))

Both a and λₐ are unknown parameters that can be estimated from past data given some prior over x, say centered at 0 with a high variance.

One can grant a new articles a 2nd look if the posterior probability of x crossing a gate keeping threshold is sufficiently high.

It seems that I'm only allowed to read if I subscribe...

Hi,

This only applies to the latest issue which is subscriber-only for 1 week. If you subscribe, the latest issue is emailed to you a few minutes later.

Earlier issues are publicly available in the archives.

Perhaps this should be clear in the main page. Something like:

> 11 Jun 2021 - Interesting Things #8 — Meta-/beta-stability (available only for subscriber until ??/??)

> 4 Jun 2021 - Interesting Things #7 — Don't read everything

> 28 May 2021 - Interesting Things #6 — Esoteric languages and sleep sort

I guess most people click in the first link and if it doesn't work assume that all the other will not work.

Noted. Thank you.

This is a very interesting collection of stories/links. I found most of the links to be very useful & thought provoking.

Thanks!

RSS feed https://bengtan.com/index.xml

Uh, this feed is for the website as a whole. Yes, the feed does include the newsletter but it's (currently) delayed by a week. Whereas subscribers get it non-delayed.

Why are you delaying the content to newsletter subscribers?

Considering the source of your content, and lack of attribution; now it's looking shifty.

> Why are you delaying the content to newsletter subscribers?

I'm not.

It's only delayed when it gets posted to the web archive. Newsletter subscribers get it straight away.

And ... (Full transparency mode) it's to encourage readers to subscribe.

> Considering the source of your content, and lack of attribution; now it's looking shifty.

Please see comments above, particularly https://news.ycombinator.com/item?id=27493014.

Right. You're talking about incentivizing a subscription to your newsletter. You have no TOS, nor PP, for it. You're providing scraped content while either time-gating it, or putting it behind a subscription. Everything together appears questionable to me.

Here's a question: If this isn't to be monetized, and this isn't about lead capture; why are you incentivizing subscriptions with current content instead of adding it to your feed?

I think it's fairly obvious. I'm gonna build a massive mailing list.

When I get enough numbers, I'll include a link to a malicious website. Then people click on it and it'll download malware onto their computer. That will seed my botnet and then I can do bad things with it.

Responding to valid concerns with sarcasm does not build trust.

To be fair, it is very possible to not currently monetize or lead capture along with little to no intention for that in the future. You just want to build an audience for the sake of it.

Of course when that power of a solid audience is actually there, then the more profit seeking intentions may come in.

Still, there’s no way to tell what path the person will go down. I don’t think assuming bad faith in your last paragraph is needed.

A trope on HN is acting holier than thou or rationalizing own behavior. IE piracy of media or ad blocking scorched earth style without true principled reason like malware or tracking.

The biggest I have found is the person likely doesn’t make their paycheck now or in the recent past from super principled pious-like work. Or they did make their money that way and are now changed.

Like The Social Dilemma film. Almost entirely filled with multi millionaires and two billionaires who made at least a chunk of their fortunes from social media. Money they are most certainly keeping and power/influence gained they are most certainly using too — I mean they are on this trending big documentary talking about helping the common man vs big social media/tech while not being the common man because of money from said industry.

Here is a feature that I'd like: curate articles by a user. Meaning if I were to, for example subscribe to all articles posted by "bengtan", then I should be able to get a list of posts that he has written, along with posts form other users that I may have subscribed to.

You can subscribe to articles posted by a user as an RSS feed: https://hnrss.org/submitted?id=bengtan

The problem though is that it is hard to find users you want to subscribe to. You need to inspect their past submissions to see if the signal-to-noise ratio of that user is high enough for you. If you oversubscribe to too many users then your subscriptions will be full of noise.

In my hobby project https://linklonk.com I want to solve this problem of discovering users and feeds that have a high signal-to-noise ratio for you.

It works similar to HN - users submit links and upvote/downvote them. But the way your ratings are used is different.

When you upvote something - you get connected to other users that upvoted the same article/post. The more shared upvotes you have with someone => the stronger you are connected to them => the more weight their other upvotes have for you.

If you downvote something - your connections to those who upvoted that item become weaker, their other upvoted have less weight for you.

Your connections to other users capture how useful their previous recommendations have been for you. And over time you get connected to the best content curators for you. All this done automatically as you rate content. You do not need to manually manage who to subscribe to/unsubscribe from.

The above mechanism works not only for users but also for RSS feeds that posted the content you upvoted. This way LinkLonk works as an RSS reader that prioritizes based on your content ratings.

We need more things like this: tools that extract valuable info from the overcrowded internet.

Agreed. I’m working on one such tool: https://trove.to/

The newsletter/sample page has a broken link: "I hope you find the following stories interesting. If you don’t, you can [unsubscribe]({{ unsubscribe_url }}) and I won’t pollute your inbox again."

Is supposed to be substituted when it is send as an email?

> Is supposed to be substituted when it is send as an email?

Yes.

I copy-and-paste directly from the newsletter content into the web archives. That's how this artefact gets transferred across.

Wow. Super interesting. I am doing something very similar, but with way less articles per newsletter. I cover 3 "snippets", which I often find on the "new" page on HN. (shoto.io)

This looks pretty good. In regards to your Substack / Buttondown comparison I would also take a look at Revue which seems to excel at curated newsletters.

Noted. I'll add to my to-do list. Thanks!

I like it, but it is so much information that you it would be cool to have a systematic way to archive it and search it.

Hmmm ... it would be extra work that would, over the long run, end up being a lot of work. I'm doubtful whether readers would search my 'archive' vs. searching google (for example).

Sweet, as someone who's never reached the front page, or the second page, I applaud this effort.

Thank you.

Is this the positive alternative to n-gate?