An empirical study of obsolete answers on Stack Overflow [pdf]
arxiv.org
arxiv.org
SO not having a good way to retire these is the same issue as twitter's 'get the facts' banner
IMO wikipedia is the only organization that's solved it, and my sense from the outside (I've only written 1 article) is that it's with editor-gatekeepers, not with tools (though there are some bots)
Easier to validate what's been carried forward, much harder to find what should've been but wasn't.
SO has crowdsourced "this is correct" (and "this is wrong") baked into its core in its voting system. One more button for "this is obsolete" would do a world of good. It wouldn't solve the whole thing -- there would be some subtlety that would be lost. But it would be at least as good as the "this is correct" signal that we already have.
Unfortunately the company has basically lost interest in its public site as a knowledge archive in the last 5-6 years and has stopped even considering problems like this, let alone working on them.
What if Wikipedia allowed articles to have two versions: the normal, policed, articles we have today - and adding a “post anything, as long as you provide citations” version for sticking all the trivia and extra details that would normally not be allowed on a main Wikipedia page.
I think there is plenty of room for all types of sites on the internet, but they don't have to all be hosted by the same group.
That's not to say there aren't rules, but rules are decided by collective decisions (or in practise the collective decisions of people from 15 years ago) and are implemented based on argument, debate and evidence. Usually when people say an "authoritarian" governance model, they mean there is a ruler or small class of elites, and whatever they say goes, no discussion, no debate, no appeal. That is very much not what wikipedia is like. It is however what a lot of social media sites are like (good luck appealing a fb moderator decision)
Typical would be; you make edits to a stub species article to add basic information from fishbase. You don't add much because you have never actually heard of the species. The next day you come back and the article is reverted or proposed for deletion as non-notable. It doesn't matter that the stub had been around for a decade, that there are 10k other stub fish articles. Nope, you tried to make a small contribution and that deserves to be punished for some reason. Eventually people get the message and stop contributing anything. Don't even get me started on trying to correct taxonomic names to the recognized name or fixing the list of species in a genus. These changes were regularly reverted without explanation. Fix 20 articles, 1 or 2 are reverted. Yep, just live with it.
Is there any way to assess editors, how good of job they're doing?
Assesing editors implies that someone is an authority above editors, to judge over them. There are of course moderators ("Administrators"), but the general idea is you are supposed to appeal to your peers and convince the group of your position, not convince someone in authority.
I'm fascinated by governance, transparency, accountability. For instance, I've been casually researching how to measure effectiveness of state legislators, how the misc rules impact the games they play.
Learning how wikipedia does things, for better or worse, is on my todo list. I'm hoping there's works like Peter Hintjens' Social Architecture for wikis. http://hintjens.com/books
Wikipedia has all sorts of internal pages documenting these things, E.g. https://meta.wikimedia.org/wiki/Wikimedia_power_structure and https://en.wikipedia.org/wiki/Wikipedia:Polling_is_not_a_sub...
Also, much of the complaints/dispute pages are very public and you can read them e.g. https://en.wikipedia.org/wiki/Wikipedia:Administrators%27_no... https://en.wikipedia.org/wiki/Wikipedia:Administrators%27_no... https://en.wikipedia.org/wiki/Wikipedia:Arbitration/Requests
After skimming these, and poking around, I haven't found any metrics for participants. Like applying Moneyball notions to wikis. A very effective, popular example is GitHub's contribution visualization.
From the hip, I imagine sparklines for various metrics. The badges are nice, but depicting the timeline is important feedback. Like how many articles an editor helped shepherd to publication over time. So it'd be more obvious if editors are gatekeeping, how many of their decisions are appealed, how many decisions they lose, etc.
Frankly, the wikipedia governance is overwhelming for a noob.
Unsatisfied with most depictions of how bills become laws, I created a graph to represent the tortured path each bill takes. (I really need to stay off HackerNews and work on my projects instead.) Imagine one of those American Gladiator course challenges as a flowchart.
Again, thank you.
So I am not sure that the guards of wikipedia is that bad. A lot of information does not make it past the guards, but it is also something which communities desire. If you don't have moderators you have upvote system. If you don't have upvote system you have algorithmic models of suggestions and recommendations. If you don't have that you have a search function that order by some kind of ranking.
I'm a correctness answerer on Stack Overflow. Top 0.14% on the site. There are established ways to overthrow correct but outdated answers (I've just used one). They have emerged organically through the community. This is mainly through necessity, especially in the JavaScript world where the standard library was missing, and now includes, nearly everything one would expect in a modern programming environment and much more.
However it would be nice for highly upvoted answers that replace existing marked correct answers that are no longer getting votes to be marked as the 'community accepted' answer.
What are they? I've tried to edit accepted answers that contain inaccuracies or outdated info and faced difficulty getting those edits approved. It is a time consuming and frustrating process in my experience.
At a minimum, that would be an input to the presentation ranking -- old, flagged items would drift to the bottom.
Long-tail "Floatsam and jetsam" content is a huge problem, generally, not just for software development information.
Might be relevant to mention that I'm quite active on the security stackexchange and regularly review the suggested edits queue (we don't have a constant backlog like stackoverflow does). Feel free to point out if you think this is not a nail for my hammer.
I tend to think the real problem is the overly strict conception of duplicate. Over time, the way people will ask a question and the way people will answer it changes.
5-10 years ago almost every JS question was a jQuery question too, now not so much. As someone who lived through that I can very easily translate to the less jQuery-centric present, but someone who started learning JS/React last week can't. A new rendition of such a question/answer would be a duplicate for me, but the old one would be obsolete to the new developer.
I think the best way forward is that both duplicate and obsolete should be soft signals rather than reasons for closing.
I see where you're coming from, but having identical questions exist alongside each other just because their dates are different does not help those who follow a link to the older question. A new mechanism to indicate different versions would have to be added for this not to be confusing, if this is the solution we want to go with.
When you search how to solve the hypothetical JavaScript problem I mentioned, the classic highly upvoted jQuery version of the question is up top, but if you can't see how that addresses your issue you can keep digging (or keep asking) for a version that makes sense to you.
Spammy, low effort, almost character for character duplicates can still be removed. I certainly acknowledge that the line is not always easy to define, but I think giving the reader what they probably want fast and then letting them sort through the long tail if needed is the approach better suited to programmer Q&A where the ability/background range is huge and database searching skills are above average.
It's true, people are so damn trigger-happy marking questions as duplicates.
I've seen new, well-posed questions on up-to-date frameworks get marked as dupe because 10 years ago someone asked a related question on an obsolete tech. The reason it was marked as dupe was simply because someone took it upon themselves to write up a sprawling smug "canonical" answer to a shitty old question that happened to cover the new subject matter.
It's much better to keep the old questions and answers, to just answer each question (and no more), and to create new questions as needed. Why not? it's not like they're running out of disk space.
I think the solution here is to encourage specific answers to specific questions, let folks sort out the historical minutiae based on timestamps and subject. Anything more elaborate is asking for mix-ups and confusion.
This seems an optimistic view. What feels an awful lot of the time it just has some of the same outside appearances, and is a different question completely, but the people marking it as dupe don’t read it carefully enough, or don’t know enough about the subject to realise the differences.
Still, it is my quixotic StackOverflow crusade and there remains more to do.
“Obsolete” isn’t a flag, it’s a version number, or even a range. This solution doesn’t work with 3.0. This one is deprecated in 3.5.
But since semver is neither universal nor infallible, you’d have to actually model languages and libraries, with a curated list of version numbers. Which is awkward when you built your entire categorization system on tagging with strings instead of modeling problem domains.
I think requiring a comment or a link to a more up-to-date answer would be nice, to avoid answers being marked as obsolete without any recourse.
The way I (not the person you replied to) imagine this is that the post goes to the bottom and gets a red background color or is faded out, similar to how deleted answers are shown in red (if you have enough reputation; in this case everyone should be able to see these) and downvoted posts are faded out.
Edit: this is the background deleted posts currently get, for those who aren't active on the SE community: https://i.stack.imgur.com/EDAIF.png
> and gets a red background color or is faded out
And most of the time, a newer answer is available (at least in my experience) so for those the downranking would also help.
1. Add an age-weighted score (and maybe sort by it) to make it easier to distinguish between an obsolete answer with hundreds of old upvotes and a recent answer with 10 recent upvotes.
2. Similar to above, but add an extra vote lever and visible score (perhaps conditionally, to older questions that stop getting upvoted?) for marking that an answer didn't work for you, or that you suspect it is obsolete, without having to downvote an answer that was given in good faith and worked for some time (and may still work for older versions/environments).
3. Add an option to open a superseding question (perhaps conditionally based on changes in absolute or age-weighted scores; perhaps it must be "community" owned but gets to inherit question upvotes) that has a special relationship with the original question and triggers extra UI on each side (to cross-link the questions, encourage users to directly mark which answers from the original work, and to provide feedback that affects when/whether the superseding question is treated as the canonical version).
Obsolete can mean a lot of different things, and there are degrees of obsolete. And people still use older versions of technology, so in some cases you might want to look for older solutions anyway. So it would likely have to be more like a version flag.
Now you need to get some people to curate that information and properly apply the version/obsolete tags. That's probably easy for some of the more often searched for posts, but very difficult for the long tail of answers. You need to educate the community on how this new feature works, and when to apply this flag. You need to decide on who can set the flag, whether you need multiple votes and how to handle disputes when people disagree or set it wrong.
If you decided that a version flag is needed, and not just an obsolete flag, you need a UI and people that manage the available versions for each programming language/framework/library.
You need to decide how to handle the same question in multiple versions. One question with multiple answers and each answer tagged with a version? A question per version? Do you actually want to enforce one variant, or allow both to exist? Questions can also be obsolete, and that is often in a more complicated way compared to answers. Should that be handled with this kind of flag as well?
This is not a trivial change, but something along these lines is probably necessary.
How do I solve problem X using version Y of Z.
In some cases people encounter older versions of technology questions and make a new answer saying updating for version X of the technology - I know this happens because I did it myself for a Gulp question and got a good number of upvotes - even though I could never actually get the accepted answer of course.
Neither of these things are true. In fact, to a user with zero reputation, StackOverflow is much more locked down than Wikipedia.
But the real issue with obsolete questions, is that a new question will be marked as a duplicate of an old obsolete question and closed before anyone has a chance to answer it.
So the real value of an obsolete tag would be that a new question couldn't be marked as a duplicate of an obsolete one.
For example, I could very well ask a new question which ends up being flagged as a duplicate of another question whose accepted answer does not work for me.
Can two questions be duplicates if they have different answers? Do I only have to claim that the accepted answer does not work to show that the new question is unique?
Wondering if there was a protocol for changing the accepted answer, I searched meta. I found the consensus is: 'Accepted' is at the sole discretion of the OP and shouldn't be mean it's correct, just that it answered the OP's question. Which I think is BS as it elevates the answer to the top of the list and gives it credibility.
Why is the OP the czar that chooses the correct solution just because they asked the question first? Honestly, the OP is often the _least_ qualified person to validate an answers correctness.
I’m not sure this matters too much for experienced devs, it’s really just a noobie trap to only look at a low upvote but chosen answer right?
Perhaps it is just a matter of properly ordering the answers, maybe giving community votes precedence over the accepted answer?
For example, here is the question I was talking about: https://stackoverflow.com/questions/11970586/apl-removing-el...
My answer should not be accepted, I was brand new to APL when I answered it. The "correct" answer not accepted and buried below two 0 vote questions.
The thing is, my answer _works_, just not well. I can easily see myself overlooking a better solution in a case like this. Maybe I'm just lazy though haha.
I'm not a big fan of the system, but just adding this to the discussion because I think it's the closest thing stackexchange has to an official way to re-open a question for answering. One of the reasons you can select for the bounty is something along the lines of "current answers are outdated".
This is a downside of gamification though: it becomes all about ego and earning points, which I don't think should be the goal of a site like SO. If keeping the quality of the content means some people must lose rep points, then so be it. If they have 14K rep they won't notice, anyway.
The flag should act so that it isn't ripe for abuse (e.g. people flagging out of spite, without evidence).
Downvoting does have the side-effect of poking the original author into action, because gamification often means (regrettably) that people do care about rep points.
edit: an argument against down/upvoting: I think there is a sizeable "bandwagon" effect. People upvote what already has upvotes, and downvote what already has downvotes. I can't prove it but I'm pretty sure this happens. If so, obsolete answers will never go away by downvoting alone.
> downvotes ... is no big deal
Some ppl feel sad and anxious about downvotes. The no big deal solution doesn't work with everyone.
Example if for the question "How to print to stdout with python", the answer " print 'foo'" is right, you just have to use python2.. it is maybe staled, outdated, but not wrong.
Both fact checking and editing old answers should lead to points; somehow.
Also maybe there is merit in adding structured data (and incentivizing it), which could be used to notify users/writers. For example adding version numbers could be a low hanging fruit.
This is just one more example of the absolutely toxic culture within Stack Overflow itself. Everyone uses it, but we all get to it from Google.
Nobody I know bothers answering questions, and I don't think more than a half-dozen people I know have even submitted questions. If you do, the gatekeepers are going to jump down your throat.
https://stackoverflow.com/users/1128957/nateeag
I've also never had a gatekeeper jump down my throat.
I think you are overstating how bad SO's culture is.
I have definitely seen problems there, but they aren't the whole of the story.
(Though SO itself has entirely lost my goodwill due to the Monica Cellio incident: https://meta.stackexchange.com/questions/342039/firing-commu... )
Honestly this is my biggest complaint with the web in general: immortal anti-information. Proposing analysis and strategies for combatting it on curated platforms is a great first step.
I think also implicit in this discussion is the role of the readers to vote with their mouses, so to speak. Without feedback from users, the mechanisms can't work effectively. Which is why I try to upvote as much as reasonable on HN and SO.
How positive are you that your knowledge is current before you vote? Users will confirm the information that they know, which is just as likely to be outdated.
Damn.
For your concern, if op said they needed help with version 2.7, presumably you'd write your answer with that in mind. Or you would say how you got it working in 3.8, and exactly how you set up your environment. If someone asked you about another version, you can throw your hands in the air and say "I don't know, but it works in 3.8 with these packages installed," and that would be a perfect response that shows others how to reproduce your work. Reproducibility should be standard practice, and you shouldn't be reliant on dubious context and dates and guesswork to reproduce an example in a website devoted to technical help.
I know this sort of feature is useful on newspaper websites - The Guardian will flag stories older than some limit as being potentially out-of-date.
That's... currently the case?
> red highlight for comments > x years
I don't find age has a 1:1 correlation with it being outdated. If some advice doesn't make sense to me, I'd look at the dates of this and other answers, because most often there will be newer answers (lower voted because they haven't existed as long / aren't seen as much) and/or comments added to the answer indicating how to do it in python3 or whatever the new thing is.
Sometimes posts from 2009 help me, sometimes posts from 2018 are outdated. Maybe this could work if a time limit is configured per tag, but even then, I expect it wouldn't be very helpful.
> That's... currently the case
Is it currently the case? I'm not seeing it, though it I may be missing it.
I do see, at the bottom of comments, something like
:: edited Apr 23 '15 at 8:40 / answered Mar 11 '09 at 21:11
The date the question was /asked/ does have an age, though, which may be what you're referring to. For the problem at hand, it's the age of the answers that matters more than the age of the questions.
> I don't find age has a 1:1 correlation with it being outdated.
No, but there is a correlation.
When you’re in the middle of work and not actively trying to learn new material I’m sure the obsolete answers are frustrating but I don’t think the outdated information is entirely useless.
But there is room for improvement.
It looks like the latest archive dump (March 2020) is available in BigQuery, e.g.: https://console.cloud.google.com/bigquery?p=fh-bigquery&d=st...
Given that SO's content is basically 100% volunteer/user generated and free to access, it seems like the first step would be to allow users to flag obsolete answers with a very visible and obvious UI element.
Maybe second would be SO fielding a team of experts as "pruners" that would delete/update the flagged answers.
Synergy!
- a fresh upvote is 1 point
- one year old upvote is 0.5 point
- a two year old upvote is 0.25 points
If nobody votes, answers relative positions stay the same.
Old answers aren't worthless but it also isn't impossible to lift updated content to the tip later.
A person might vote for the same answer again after half a year, but it really just refreshes the vote to 1 instead of adding more value to it.
I'm not a fan of deletionism and in stack overflows case I'm fairly certain it has destroyed value for millions both in knowledge and reputation.
(Why? For years around 2009 - 2015 or something whenever you found a good answer that solved your problem it would most probably be flagged for removal or something.)
They will also not allow duplicates, allow low effort questions - even if others are lining up to answers those low quality questions.
Whoever wants to improve the world as much as stackoverflow once did - and make a bunch of cash in the process could try some of thise ideas.
Someone will probably counter with the assumed fact that if there had been money in it someone would have done to which I counter with the two economists who went down the street, saw some money on the ground and walked straight past it since "if the money was real someone would have taken it already".
Still in the early stages, but it's promising.
The actual instances were somewhat hidden as far as I could see.
But. I am more interested in the inevitable corollarial follow-up:
An Thorough & Ignominious Probing Investigation Into The Problematic Presentation Of Pseudo-Intellectual Wankery On Hacker News