Would you be willing to consider the possibility of delisting the site with robots.txt?
HN is an experiment. As a rule, a community site that becomes popular will decline in quality. Our hypothesis is that this is not inevitable—that by making a conscious effort to resist decline, we can keep it from happening. - https://news.ycombinator.com/newswelcome.html
pg wrote that over 15 years ago. I've been saying for (a mere) 10 years that we're trying to stave off the arrow of internet entropy: https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que... – not doable forever, no doubt, but doable for a while if we expend enough energy.
Based on experience so far, the way to do this is through a combination of a dedicated community, software that does what software can, and moderation to bump the system out of its failure modes.
Trying to hide from the outside world is mostly not the way. pg used to do tricks like Erlang Day on occasion, but having that as the main strategy would be like trying to avoid infection by never going outside. Far better is to have a robust immune system, if possible. Trying to avoid (resistible) pathogens weakens the immune system, and hiding HN from new users is a path to decrepitude. The latter is probably a greater threat IMO, and too easy for established users to discount.
Spam, and its cousins like content marketing, could kill HN if it became orders of magnitude greater—but from my perspective, it isn't the hardest problem on HN. That's because of the dedicated community, which flags these things, reports them when they escape in the wild, and is vigilant about quality. Without such a community, HN would have died long ago.
By far the harder problem, from my perspective, is low-quality comments, and I don't mean by bad actors—the community is pretty good about flagging and reporting those; I mean lame and/or mean comments by otherwise good users who don't intend to and don't realize they're doing that. There's an unholy dynamic between those and the upvote system, so worse comments often get upvoted more than better comments do—often enough to choke the threads with weeds. That's the high-order bit and what I spend more time worrying about—not Cassandra E Oakley and her trading system*, nor the latest startup voting ring and whatnot. If those ever become the high bit, we might be doomed, but we should see it happening long enough in advance to readjust.
p.s. (@dang doesn't work - I only saw this by accident. Well, not by accident because it ended up at the top of the thread)
* https://news.ycombinator.com/item?id=39425371 - unkilled for the occasion
blame those dang programmers!
I agree it probably doesn't make sense to try to block growth sources though. Just considering the perspective "it's not effective to try to play whack a mole anyways" already seems enough evidence of that alone.
All this reminds me though I've been using the site too much lately and my interaction quality has gone down. Time for another short break :). Anyways
So many places on the internet used to be wonderful places of discussion. I remember as a teenager I would come to the internet and be in awe at this system that mankind created to allow for all humans to come together and discuss.
But I’ve noticed since 2015 every single discussion place I used to frequent online has become horrible.
I discovered hacker news only recently and this website seems like the last remaining jewel of the internet that still exists. I think the reason is because of you and the philosophy you use to moderate the website.
Just wanted to express my gratitude to you. Not sure how often people say thank you to you for the work you do but they should say it more often!
At first I thought it was a site entirely devoted to computer stuff but it was only later that I realized what this site was really about.
Can you elaborate on this a bit? I don't see why "lame and/or mean comments by otherwise good users who don't intend to and don't realize they're doing that" should have an "unholy dynamic" with the upvote system.
(FWIW, what I have observed is that once a comment becomes established as the top comment in a thread -- and it doesn't take much for that to happen -- it is nearly impossible to dislodge it. That means that getting into a thread early is crucial for getting noticed. I've pretty much stopped commenting on threads that are older than an hour or two because I can be 99.9% certain that whatever I write will never been noticed no matter how good it might be. And FWIW2, the comment I'm responding to is 50 minutes old as I write this.)
Sometimes moderators actively bump down top comments.
It’s probably related to what dang wrote above about the “unholy dynamic” of low-content comments that trigger easy upvotes. I remember myself having written some (frankly) lightweight low-effort snark, see it end up at the top of the comments for a few hours, and then it went mysteriously halfway down the list even though the upvotes didn’t stop.
To be clear, I think it’s a good thing the moderators do this kind of weighting, and the invisibility of comment upvote counts to non-authors is an important feature because it enables this.
People optimizing for this kind of low effort but highly upvoted comment is called “karma whoring” on some places.
The reflective circuitry, which takes in new information, turns it over, and generates an unpredictable response, is much slower and harder to run. I suppose it's a bit like the difference between a sugar hit and eating nutritious food with fiber. The latter makes you feel better in the long run, but when it comes to mass dynamics, the sugar hit wins out every time.
> once a comment becomes established as the top comment in a thread -- and it doesn't take much for that to happen -- it is nearly impossible to dislodge it
Moderators downweight top subthreads that are generic or otherwise lame, and repeat this until the top subthread is no longer lame—if possible. The trouble is that this is an intensive manual process. Most likely the software needs to be adjusted as well.
[1] https://hn.algolia.com/?dateRange=all&page=0&prefix=true&sor...
[2] This is probably the basis for the generic subthreads which are the bane of this forum: not bad enough to flag, but predictable enough to suffocate.
context: i read the "past" page every day, so often am several hours or over a day behind discourse. interstingly enough this post is still relatively new and was popular enough before midnight utc to show up in today's past page.
i wonder how much positive feedback posts receive if they miss the initial window.
I know HN does something similar, but it is not quite as good as reddit. From observation, specifically the 'penalty', or added performance requirement, of latching on to a top post is too weak. The result is that all HN comment threads consist of only a few top level posts, with subthreads growing off them, because you can easily 'jump the queue' just by commenting to a top comment. This is also what contributes to the idea that it is pointless to make new root level comments after an hour - because almost all the action is in subcomments to top comments.
Edited to add: Reddit soft-hides (collapses) subthreads that are deemed lower importance, which is probably key to make the ranking system work. Anyone interested in a subthread may expand the hidden/collapsed sections, and they may even be upvoted back to uncollapsed state. But by default they don't muscle into the main conversation. HN already has the collapse feature, which could be reused for this. It's just a client-side collapse, also the reddit one (though in huge threads, deeper threads will be loaded on demand).
How?
> the 'penalty', or added performance requirement, of latching on to a top post is too weak
I'm confused by what you mean by 'quality' and 'performance', unless you just mean upvotes.
I think reddit just counts a ratio of upvotes to views (ofc downvotes too). It is possible that users collapsing a comment/subthread also has some weight. Would make sense.
> And, importantly, the quality requirement for the post increases the higher in the total hierarchy it is: root-level post, top-level reply to a top comment, and so on
What I mean by this, is that the more prominent a place a post holds, the better it must 'perform' (per the above definition). Prominence being mostly just how high on the page it is.
considering this along with the comment below in the thread:
>most activity on a thread seems to happen in the first 24 hours of posting it. the discussion tapers off beyond that.
Supports more software experiments to encourage "slower" or longer discourse, and thus reflection.
This is my longtime wish as a user: reading via RSS most conversations are long dead when I get to them. I subscribe to replies on my comments, but since few other people do, it's not that useful. Simply promoting such features might help, as I bet few users are aware of them.
1) my comment actually did capture your attention 2) I did notice it (courtesy of HN Replies)
The current state isn't working too poorly. and sure beats the pants off any other option i know of ;)
So please keep up the awesome work.
And the other problem is that even when it works perfectly, this process as described can only produce non-lameness at best.
I have a suggestion based on something I did at Google 20+ years ago: compute Page Rank on commenters. I did this 24 years ago in the Google Translation Console, which was a (now long-since-retired) public interface for volunteers to translation Google's site content into other languages. Translators would not just input their own translations but also rate the translations submitted by others. Translators whose translations were upvoted more often were deemed more reliable raters, just as links from highly ranked web pages are weighted more highly when computing Page Rank. This successfully prevented any translation spam from ever making it through to the public site (as far as we could tell). It never reached the scale of HN, but it's a lot easier to implement such a thing nowadays too, so I think it might be worth a try.
How do you structure this to avoid that problem? How would the system work for a new account just created to get established?
Just out of curiosity, who would you cite as an example of a lame commenter with tends of thousands of karma points?
Also, stuff like Erlang day isn’t a bad idea: it is also a kind of nudge to remind the community about what kinds of discussions and comments to prioritize.
> This ratio has steadily gotten worse over the past years
I'm not sure. Everything has always been getting worse, or feels like it has. You'd have to discount for this bias in order to tell whether things have really gotten worse, and no one really knows how to do that, nor really wants to. It's more satisfying to feel the decline of civilization setting in as one ages. Yes, that's a grandiose generalization and therefore lame.
Or, to flip it into a positive, you're more experienced and informed than you used to be, so more comments appear misinformed or inexperienced.
This is a problem I've seen in specialist forums elsewhere; the conversation runs out, because nobody's poking the community with wrong stuff, and correcting a wrong statement is intensely more motivating than just making a plain factual statement for its own benefit (also: when you do that, the rest of the community will go "uh, yeah, and?").
Would be great to see some actual data analysis.
I wonder if that would give enough pause for users to consider if a comment is worth their upvote.
However, while I completely agree that spam isn't anywhere close to being a threat to Hacker News, I'm not so sure about subtle, well-executed content marketing/influence campaigns. By design, these are much less overt than mere spam links to a porn website, and there are plenty of places for them to pop up on HN - for instance, in any number of recent threads lamenting the sad state of modern appliances/tools (like vacuum cleaners, e.g. [1] - and it would be extremely difficult to even approximate the scale of the problem.
I'm not advocating hiding from the outside world in the same sense that one might try to intentionally isolate their community to prevent others from joining - just in the sense of avoiding a specific discovery mechanism (Google) that results in extremely strong incentives for content marketing companies to infiltrate and influence. More specifically, I posit that the amount of organic discovery and community growth that we would get would be significantly outweighed by the attention from content marketers - most people that I know discovered HN through blog posts and individual recommendations, and not through search engine results.
Ultimately, of course, it's up to you - but please think about this issue (and particularly the tradeoff of the upsides vs downsides of being highly-ranked in Google results), and look for indicators of content marketing. Hacker News has, by far, the highest quality-size product of any community that I've seen on the internet ever, and I would dearly love for it to stay that way for many decades to come - which I don't think will happen if "Hacker News product recommendations" becomes a currency with supply comparable to GitHub stars[2].
Similarly to how reply is hidden for the first few minutes for hot threads.
(BTW, if this is gonna be done, we should also block all AI bots and search engines at the IP address level.)
That's not possible or necessary.
Not possible, because it simply isn't possible in the general case to differentiate real papers from scrapers, without using device attestation. For an extreme example of this, see the Recap the Law project[1], which gives real human users an extension which scrapes as they browse.
Not necessary, because scrapers for AI training data an entirely separate problem completely unrelated to marketing, and because robots.txt will serve to stop the majority of search engine indexing, which is all that we need. Actual blocking of engines isn't necessary, because all the big ones are well-behaved, and after they stop indexing HN, marketers won't care about HN for the purposes of SEO-related influence campaigns any more.
Email it is!
The linked blog post actually describes that Google is drowning in ad-driven crap, and has to rely on manually chosen external structures to keep quality tolerable, not that “HN has became cool”.
Profiling other users, in addition to being extremely bad form, is a pretty good sign that you don't actually have a valid argument to make - which, reading your comment, you don't.
My language was "Would you be willing to consider the possibility" - a polite request. There's no way that a logical person could even read that as either a command or like I was acting as "the voice of the majority" or "a plurality of users" - you pulled that out of thin air
Dang is not a tool or an idiot - if he thinks something is a bad idea, he won't do it. If he thinks that it might be worth doing, he'll think about it. If he needs to poll users, he will.
My entire life it's been my experience that the less people involved the better things are in general. There's probably an optimal number of people for every community. I don't really know what that number is but I seriously doubt it's in anyone's best interests to exceed it.
> I would like to make the argument
Which makes this perspective very confusing:
> it seem like you were the voice of the community
Browsing HN is a relief in times of Surveillance Capitalism
Edit: HN uses google tagmanager and analytics; amplitude and branch.io
We do? No we don't? I don't even know what those are.
If you see requests outside of that there's something fishy going on in your browser, I believe.