The Fall of StackOverflow: A Data-Driven Analysis
pdftranslate.ai
pdftranslate.ai
But why? I ran a site called WPQuestions.com for 7 years, it was where people paid between $10 to $50 to get fast answers for their WordPress questions. I called it "micro consulting" in that the consultants were answering small questions for small fees. While I get that StackOverflow wanted to be free, I wonder why it didn't commit to being more of a Helpdesk for people in need. Why have general answers, when people have such an urgent need for specific answers to their specific questions? An incredibly wasted opportunity. Even more so, with AI able to answer all of the general questions, the only avenue open to non-AI companies is answering very specific questions that are heavily context dependent.
Stack Overflow sort of flirted with this idea, I guess, with the concept of attaching "bounties" to your questions. The bounties were just Stack Overflow points, not cash. But AFAIK that wouldn't spare your question from being marked as dupe so you still couldn't use SO for support.
The 'dated' user interface is irrelevant, people don't scroll it endlessly like they do on facebook. they come for question and answer, the ui is fast and well made for that purpose.
The UX not so much though.
There's so many x.stackexchange.com estates that often overlap (devops, serverfault, linux, askubuntu etc).
The search is, and has always been, very poor to unusable. I've always just reverted to DDG or google to search stackoverflow.
Having loads of answers and questions in a clean and fast UI is good, but if no-one can actually find the right Q and A, the overall "interface" is just outdated and poor.
(Edit: related: I've always voiced the opinion that "duplicates" aren't solved with more or harder moderation, but with better search)
Yes, but it's implemented in a user-hostile manner that kills engagement and therefore retention.
A friendlier implementation imo, would be for the asker to be shown the duplicated answer(s) inline under their question, with a note that it's a dupe and a link to the original question(s). That leaves them with a vastly different experience, and regular users can still continue to see it as just closed. Kinda like shadow banning, but healthier.
That would occupy a lot of space. What's wrong with the current approach of posting a link to the duplicate? Mind you, when the first user votes to close as a duplicate, the system automatically posts a comment with a text more or less like "this question already has an answer on this link http://stackoverflow.com/.... If you think this case is different please state why". Then after three votes the question gets closed.
But please notice it can still be reopened. Users with 3000+ reputations (so not only what many people percieve as sort of barons on stackoverflow) can vote to reopen. And three votes are enough to reopen a question.
Simple. The user's problem is not solved. The whole point of StackOverflow was to help people get answers to their questions.
If you fail at this, you lose your relevance.
>>Users with 3000+ reputations (so not only what many people percieve as sort of barons on stackoverflow) can vote to reopen. And three votes are enough to reopen a question.
You don't have to under go these rituals in front of LLMs just to get your problem solved.
Even until now, StackOverflow had no competition, so it didn't matter. Now that there are alternatives. People won't put up with user hostile experience.
SEO the hell out of the "pile pages" so the top hits go to something useful, as opposed to a closed question with everyone commenting that it's the top Google result.
Closing actual dupes? yes. Closing as "dupe" questions that sound similar to other questions, that's a massive bug, and happens if "close all dupes" is a major goal for people, over "ensure I understand the question and the question I think it's a a dupe of" (which can be quite hard and specific!), the overall context (yes, the question was answered in 2009 with "use jQuery". No, that's not the answer in 2024, but there is no process to "re-open" a question where the state of the art has changed), ...
Of course there is. When you get closing privileges you also get re-open privileges. And, believe it or not, people do reopen questions when it's appropriate.
the overall context (yes, the question was
answered in 2009 with "use jQuery". No, that's
not the answer in 2024
At a minimum, for that problem to be overcome, IMO there would need to be a lot of metadata associated with each question so that moderators could make faster and more accurate judgement calls regarding the question of, "is question A a dupe of question B? are question B's answers still relevant?"
This seems like an issue where LLMs could actually be of some assistance. In my experience ChatGPT has been pretty good about understanding that sort of change over time and which solutions are deprecated. I wouldn't want LLMs making the decisions outright, but I could see them being a valuable assist for human moderators.
But I don't think their is a practical upper bound on good questions though, so the reduction in total questions and answers in those graphs is definitely not good.
I have always thought the scale itself is very hard to maintain. I thought the different sites had a good balance but it is hard work. Mods and superusers should get compensation.
The problem is with uneducated mods who falsely close a question as a dupe.
Separating real dupes from possibles dupes is a hard task, but if they are not up to the task then SO needed to find people that were or change the system.
there are only so many questions you can ask about e.g.
how to write a CSS selector before all the others are
duplicates
Think how many ways there have been to center a freaking HTML element over the years! There are, at a minimum, at least 10-20 "correct" answers to that question over the years.
Even today, that answer varies a lot. What is your render target? A state of the art desktop browser with automatic updates turned on? Browser in some locked down corporate/government/military environment that is 5 or 10 or 20 years old? HTML embedded in email? Some combination of those?
But Stack Overflow's design lacks that kind of specificity - there's no metadata associated with questions that would reliably specify the specific environment and software/library versions being targeted.
And even when the person asking that question does give that information, the moderators don't have the time or the inclination to manually and laboriously parse out those specifics and compare them to the specifics of the other possible dupe candidates... a comparison space which grows by zillions of candidates per hour.
Instead of striving to improve the user experience then the moderators spend their time bickering over meta issues. Many haven't asked or answered a question in the last five years.
1. Post a question as a new user
2. It goes into Staging Grounds instead of the main site, so there's no way to get an answer
3. Watch as a handful of powerusers flag it and demand changes, with a combination of user-written and automated comments
4. Make some of the recommended changes
5. It gets closed anyway and never makes it to the main site
To me it seems like a forum that effectively sanctions the bullying of new users. I'd like to see Staging Grounds success statistics.
Thinking about it, know I want a "Current Drama" button where the "I'm feeling lucky" button used to be on gSearch :)
And of course many people come to SO with unformed questions or want a discussion for the social interaction, resulting in them getting hit with the SO hammer. Not quite sure where they've ended up, but the obvious one is Reddit.
People keep mentioning Discord, but I've no idea how you'd actually use a programming discord in the same way as you'd use SO, because if you make a post and come back an hour later surely it'll have vanished into the flow?
At least I feel more often the need to ask if I'm making a sane choice for the next few years rather than literal `1 + 1` questions with clear ever-green answers, which StackOverflow seems to prefer.
Otherwise, if you explain why you’re asking the question again, link to the previous question, and maybe present arguments why you think the old answers might be outdated, then it’s lesa likely to be closed as a duplicate.
[1] https://data.stackexchange.com/stackoverflow/revision/185820...
Behold this X post, and cringe as you slowly die... https://x.com/dagelf/status/1827622956849078618
…if so, it was published a year ago: https://www.linkedin.com/posts/erik-h%C3%A4nel-6a44151a6_num...
*EDIT:* The linked site (with an ironic… or apt domain-name) finally loaded for me - it is a different article.
Interesting to compare conclusions though:
2023 article:
> In summary, it appears that the biggest contributor to "StackOverflow's fall" is not one of the two LLMs GPT3 or GPT4, but rather the market power of Google's algorithm. The latter's changes in May 2022 caused a significant drop in traffic that is much more significant than the effect of GPT. However, since traffic also provides advertising revenue, this can be a big problem in the future. […] Of course, this also reflects the previous analysis: most of the generally valid questions have already been answered (and their answers can also be generated by GPT). What remains are the few questions that are very specific.
2024 article:
> StackOverflow's decline was a result of issues within its model and execution, which predated ChatGPT's introduction. While ChatGPT may have accelerated this decline, it was not the sole cause. There are likely many factors contributing to StackOverflow's struggles, but AI alone cannot account for the entirety of its challenges.
They both kinda agree - if you squint a bit.
Discord is a -ing chat platform. They haven't fixed anything for you. The various communities that run their own discord based chats may or may not have fixed things for you.
Yes. This is one of the key reasons behind creating SO and why it worked so well at first.
As well as non-current content in some sources getting buried or otherwise being hard to find, or simply expiring (individual forums going offline), the other issue was the distribution of people willing+able to answer questions around the many disparate forums was inefficient – for all the talk of “the Internet should be as decentralised as possible” no one has found a better answer for this than a bit of centralisation with open licensing (SO wouldn't have attracted as many of the better people answering if their work was going to be locked in rather than covered by something like CC-BY-SA).
The solution that is taking over a bit ATM is Chatty Glorified Predictive Text and its friends, because this gets around the latency issues (you get a much faster answer, even if an incorrect one that means you need to rephrase the question several times) and licensing issues (though IMO the morality of that is rather dubious, the legality of it is still being argued in a number of places, but nether question of legality nor morality is going to stop it happening going forward).
My experience was that SO excelled at answering basic questions with definitive answers, but anything creeping outside that sweet spot would be poorly understood by both answerers and moderators. It’s still a useful resource for that historical record of basic answers, but it feels like complex, niche or subtle questions are not welcome.
The view count sharply dropping off around 2013 mirrors (to me) the increasingly more narrowly defined scope of the site's acceptable content than what it was when it started.
Digging through my local notes and journals as to why I stopped, I have an unpublished draft blog post from 2012 that has some examples of bad questions from 2012, and I later added https://meta.stackoverflow.com/questions/252506 to it as something that resonated with me - "Why did I even help this guy? I have just been wasting my time". I must have been burned out on the whole thing by that point, but it was 10+ years ago and I've not even visited SO in years.
I do remember they added a tier of moderation around this time, and I had sufficient points to have access to that. Every time I visited the site it showed a red dot on the stuff-to-do icon, encouraging me to review a queue of bad questions. I'm pretty sure that was when I decided to stop contributing, because the site felt like it was more focused on the tedious bureaucracy of it all than the technical stuff.
One of them changed so that when you reviewed something it would show you at random a sample that had already been reviewed, but this was not indicated in any way. If you got the review "correct" it said "well done, that was to test you are paying attention!" - it was even more patronizing if you got it wrong.
I was so annoyed that they had wasted my time in this way that I stopped contributing all together. It was a wake up call that the whole site was taking advantage of our free labor, and I decided to do better things with my time.
When someone asks “how do I X” and the only answer is “do Y”, just saying “don't do Y” is not helpful because that amounts to “don't do X” and the original asker might not have/want that option.
They have to be asked in the sort of detail that most of those asking do not otherwise any answers end up either wrong (because of details not known at the time of writing) or at least as vague as the question. Also, a long question will put people off actually interacting with it which is one of the reasons people aren't going to change how they ask the question, that and sometimes people don't realise that their question isn't simple (though why they don't realise that when they can't search for a simple answer seems like an odd disconnect to me!).
Diminishing returns.
A 100% successful StackOverflow SHOULD show a tapering off of questions. This is exactly what we should expect from perfect success. Now, I don't think perfect success is what we see, but a system where a question can be answered once and then re-read many times should absolutely not keep new question counts high, that would be a sign of a massive failure.
The view counts is harder to explain. I would guess it can be partially explained by search engines inlining stackoverflow answers into the search results so people DO get answers from SO, but don't actually go to the site.
The result for me was that I started to rely much more on documentation and writing down notes of stuff I found myself looking up a lot. I never used SO directly for looking up stuff, always relied on search engines, and as they declined in quality I just stopped visiting SO almost alltogether.
I imagine I'm not the only one with this story. Considering users also abandoned the site for LLM assistans and the strict moderation on there, I'm not surprised it is in heavy decline.
This seems to be wrong. According to the graph StackOverflow has nearly no views at all, which can't be right. The link goes to a query that selects votes not views, and does indeed show an enormous decline. But it's possible that people still view the site, they just don't vote.
This is happening already with Discord communities but it's not searchable or usable and the format is not good.
StackOverflow (and mods on a power trip) betrayed the community. I hope something fills the void. I'd be happy to help or even throw a few bucks in. Or at least contribute. Maybe this time not VC-backed nonsense.
- gamification problems: e.g. a race to answer instead of thinking it through, downvoting competing answers
- the accepted answer sometimes is incorrect (but sounds good enough to OP)
- balance between repeat or vague questions and over-zealous mods blocking
- need of better mod vetting (ideally, paid)
- question quality
Maybe all answers should be wiki-style (cooperative).
Maybe questions could be first in a staging area and meta-evaluated if it's good, dupe, or too vague. And it could also be improved by the community.
And some mechanism to avoid Eternal September!
It’s valuable to read different perspectives, and even conflicting opinions, and have them attributed. Having only a cooperative answer would lead to edit wars and would drown out minority perspectives.
1) a lot of questions have been answered. How often do you ask something that is answered by a 2010-2014 answer very well? IME, often
2) programming isn’t as niche as it used to be. these days, a lot of SO is just asking for free labour on very niche questions like an exception someone is getting. which was never the point of SO
there’s a wealth of answers on SO historically, which are great. But I don’t remember the last time I used SO to answer a specific question for a recent software tool, or relied on any post-2021 question/answer
I vividly remember asking a question before was different in a subtle but important way from existing ones (which I'd seen) and it got marked as a duplicate.
Also your answer sounds like your average SO moderator's :)
I, for example, frequent almost exclusively the Java tag and I can assure you that the question we mark as duplicates the most are very few:
- How to compare strings in java (because people keep using == instead of `.equals()`
- Why doesn't this class/array change when I pass it to a method (because java is pass-by-value, not pass-by-reference)[1]
- What does "cannot find symbol" mean (where the duplicate target helpfully explain in what cases this could happen, including, but not limited to "you declared it inside a loop so it's not visible outside) [2]
- Why my scanner skips reading a line in this loop (because every time you use `nextFoo` where `Foo` is not `Line` you need to read the newline as well before continuing)[3]
The other sort of questions we tend to close are the one easily answerable by learning any basic tutorial, the ones that tell us "my code doesn't work" without showing their code, the ones that just dump their homework and expect us to write a solution. And casual user rarely see this questions because we usually also delete them, but I can assure that most of our moderation time is spent handling this kind of stuff, not closing as duplicate.
On the other hand, one kind of question that is not well received but IMHO should not lead to closing or downvoting (but it does) is "I want to do this weird and non-idiomatic thing. How do I do it}". Unless, of course, it's an actual XY problem, in which case it helps if the person asking the question states why they want to do such a thing.
All of this to say, your experience with the site will vary depending on what technology you're asking about.
And "your answer sounds like your average SO moderator's" yes, it does, because we use the site a lot and we know what we're talking about :)
[0] https://stackoverflow.com/questions/513832/how-do-i-compare-...
[1] https://stackoverflow.com/questions/40480/is-java-pass-by-re...
[2] https://stackoverflow.com/questions/25706216/what-does-a-can...
[3] https://stackoverflow.com/questions/13102045/scanner-is-skip...
I can assure you that when I end up on SO I somehow run into questions like I described. Or I have a problem like I described but I don't bother asking a SO question because I know the mods will descend on me like vultures :)
SO moderation may be good for beginner questions like you describe, but I'm past those and my problems tend to be pretty subtle.
> your experience with the site will vary
Well my experience with the site tends to be zero these days, unless a search brings me there. I used to look for questions to answer in my areas of expertise on my morning coffee, but then the push for ready to copy/paste answers happened and my answers that tried to get the asker to think for themselves weren't good any more.
Additionally, isn't the goal of a site like SO is to finally answer every programming question? It is not an insurmountable task if you disallow opinionated questions and be strict about duplicates.
I feel like it serves its purpose well and will be a valuable resource for years to come.
There are some niches sites in the network like bicycling, English, Mathematica, Academia, etc. They contain extremely high quality answers. But for programming, it's nowadays much more easier on GitHub and relevant issue trackers. I think it's good, because there are many questions in those sites that contain often outdated answers. Some lucky questions get a comment there saying the answer is outdated, but there are many outdated answers.
Perhaps, JavaScript has a lot to do with this. JS evolves at a much faster rate than sites like StackOverflow can keep up.
Obviously people asking LLMs instead is an important factor, as are unverified (often not entirely correct) answers from those using LLMs to gain Internet points.
Que? What was sold - by whom and to who?
StackExchange is still its own company - and owns StackOverflow - doesn’t it?
Fortunately community-moderation works well - for the most-part - on SO/SE; not to say SE is anything like Reddit (e.g. the impotent protests over “Monica” which feels like it happened over a decade ago - and people still reference that episode in their display-names).
If I ever become a moderator on SO it’ll be because I want to bolster my resume (I did get to interview at BlueOrigin on the basis of my SO karma score - so it has value; certainly more than LinkedIn, ew)
They do a different kind of moderation. For example we community moderators can't delete posts unless certain conditions are met (a certain number of downvotes). And we can't send warnings to user when such a warning is warranted.
Also I think they have access to some somewhat sensitive information that you may not want the whole community of high-score users to see without making everyone sign some sort of nda first.
Other users did vote for them. There are actual elections for diamond mods. The fact that you didn't vote for them doesn't mean they don't have legitimacy.
> I generally don't trust anyone who wants that kind of power online for no compensation.
As a community moderator I have a lot of power over certain things (I can unilaterally close Java questions as duplicate, I can edit questions, vote to close, vote to delete...) and I do it because I like the website to stay clean. Why would you doubt their motives just because they can do a bit more than what I can do? Do you distrust forum/subreddit/discord moderators as well?
> Other users did vote for them. There are actual elections for diamond mods.
I don't like the idea that I have to be on board with a website's unvetted volunteer moderators in order to use it.
I agree with SO but I don't agree with its power mods. The power mods are constantly at odds with SO's policies. Therefore it's a power mod issue and not mine.
> Do you distrust forum/subreddit/discord moderators as well?
Absolutely! Why should I trust these people? Some subreddit moderators for specific smaller niche subs can be ok, but a lot of questions should be asked of large subreddit moderators.
Who are these people who spend hours a day working for free for a billion dollar company? Why do they do it? Why are they moderating 100+ subreddits? Why do all of their posts automatically go to the top of the Reddit front page even if they get relatively few comments?
Reddit seems extremely easy to astroturf and it's very clear to see during US election periods like we're in at the moment.
I get quality answers on IRC. Most other places seem flooded.
(Admittedly, 20! is a big number.)
With all sincerity, I honestly don’t know what this is supposed to mean.
Right, that was one possibility… my initial thought was that you meant HDD capacity.
But even with that clarification, how is an answerable question? Are you asking people to guess (Feynman Approximation?) how much internet traffic up+down that the average MacBook user has per-day compared to Windows or Linux users? Or did you mean something else?
My left eyebrow raised itself uncontrollably as I typed that out, in fact.
Exactly, you did not. How I see it:
The problem with reading such complaints is that we always only hear one side. That's why posting what you actually asked is valuable information to be able to say whether your complaint is justified or not.
By default, a lot of people immediately side with and upvote the person writing the complaint, but that's actually bad - as I said, we only hear one side of the story that way, and in cases such as this one what was actually said/asked matters and without that piece of information any opinion one forms and comment votes one gives do not have a good basis to stand on.
Furthermore, when people write such complaints but don't see how from the reader's PoV is only biased half-information I wonder if their question showed a similar lack of how it looks like from the PoV of its readers.
You want me to side with you, but you withhold information that would let me evaluate if you are actually in the right.
I don’t do that (downvoting, that is).
No-one I know does that.
It sounds like you had a bad day.
Comment chains on HN and elsewhere are not private one-on-one conversations.
Not accepting that things work the way they do, does not stop things working the way they do.
i know people here like to hate on discord servers, but this is one thing they shine at, it's easy to drop in and just ask a damn simple question to people that are online, in real time
Not realtime, sure, but for me most of that time that's perfectly alright.
For specific questions related to a particular technology: look for the specific Discourse (or forum equivalent) site.
Otherwise, discord as last resort.
And yes, the usual thing: since it's not a forum it's impossible to search the history (yes they have a search button which is useless for howto type questions). It's also impossible to browse it like a forum to see what problems people have encountered before you have them and possibly avoid them, or get better insight in the stuff you're using.
The value of public forums is that their content appears in public searches, can be publicly linked to, and that they facilitate asynchronous, longer-form and longer-term discourse.
SO is still the best evergreen forum for a bunch of things. Not to mention, how much dumber would today's almost-not-dumb AI's be without its data?! Not everyone should measure their "health" in terms of foot traffic. "The best things on the internet can't be measured?" ... a modern counterpart to "can't buy me love?"
... or answering questions.
If you answer with anything but something ready to copy/paste the moderators will frown upon it at best. No "teach the man how to fish" answers.
To rephrase in 2024 terms:
Before LLMs SO's moderation discouraged questions that a modern LLM couldn't answer (anything going into more than syntax) and also discouraged answers that are beyond what a modern LLM can generate (i.e. code ready to copy/paste, possibly wrong because neither the LLM nor the reputation chaser bothered to at least try to compile it).
But let's not blame just the moderation. Karma whores that answer every new question with a mildly related copy/paste within seconds have been a thing ever since SO grew up.
Or is this a first-derivative of [answer/question] count ie the rate of change? In which case a) this should be clearly stated ("questions added per period x"), and also, it wouldn't be surprising that the charts go down because the knowledge is already there and so fewer questions/answers are needed, even without AI.
While there is indeed a big decline post-GPT et al, the decline started a lot earlier than general availability of AI.