Ask HN: What if Stack Overflow suddenly decides to shut down? | Hacker News Reader

Ask HN: What if Stack Overflow suddenly decides to shut down?

54 pointssyx78 comments

As the title says, what would happen if Stack Overflow decides to interrupt the service? Realistically speaking how much damage could that do to everyday work?

View on HN

The US intelligence community actually keeps a clone of Stack Overflow that is updated every 12 hours and copied to servers on the classified version of the Internet so that people developing classified software can use it without needing to turn around to their unclassified workstations. I'm sure they're not alone in this, so the entirety of the answers there could be restored from mirrors or just used from the mirrors if the main site disappeared.

Nearly all of the information there is also available in the public documentation of whatever the question asker is asking about anyway. Forcing developers to use real documentation instead of Stack Overflow would not likely hurt everyday work. It might become harder to find literal worked examples, but even those are mostly duplicated elsewhere and SO is just making it easier on search engines to find it.

> Forcing developers to use real documentation instead of Stack Overflow would not likely hurt everyday work.

For beginner-level stuff maybe, but there's so much on StackOverflow about bugs, pitfalls, caveats, breaking changes, rationales, use cases, background, etc. that you can't find easily (if at all) in software documentation, if the documentation even pops up in your searches.

> there's so much on StackOverflow about bugs, pitfalls, caveats, breaking changes, rationales, use cases, background, etc

this question is closed as off-topic.

How do you know that info in the first paragraph?

On top of that security, it'd also be interesting to monitor what someone (based on their IP and browser fingerprint) was looking up on SO, you could probably figure out what they were working on that way.

Although I guess when you work for the spooks, they could figure out how to mask your IP that each query would come from a different IP from the world (basically I'm describing Tor, I wonder if they have their own Tor network).

> On top of that security, it'd also be interesting to monitor what someone (based on their IP and browser fingerprint) was looking up on SO, you could probably figure out what they were working on that way.

When the security incident happened last year on SO, SO team monitored what information the attacker searched in SO. So I guess some people want some privacy in searching information in SO.

https://stackoverflow.blog/2021/01/25/a-deeper-dive-into-our...

The second part, regarding mirrors, https://archive.org/details/stackexchange

> Nearly all of the information there is also available in the public documentation of whatever the question asker is asking about anyway.

In the absolute that may be true, but even if a low percentage of those are not available elsewhere, they are absolutely crucial. At least, they have been crucial to me.

I'm too young to know if it's a recent development, but documentation this day ain't worth much.

Oh, no; it has always been that way. Either too much information (needle in haystack), or undocumented (think Apple's latest Swift). The best documentation I've seen is the PHP site that not only has all function listed, but gives official examples, with user comments that can be voted on.

> Nearly all of the information there is also available in the public documentation of whatever the question asker is asking about anyway.

That's really not true in any meaningful way. Maybe the information is somewhere, but some Stackoverflow answers are so good that they are the best resource on a specific narrow topic that you will find online.

See for example something like [0]. That's the type of answer you would get from a good book or teacher, good luck finding something comparable in the often terse official python docs.

Sure, there are a lot of lazy SO Q/As that sound like someone wasn't doing their homework, but that's far from the only type of content there.

[0] https://stackoverflow.com/questions/100003/what-are-metaclas...

(Edit: another great example is basically anything that concerns advanced usage of matplotlib. That library is so weird and the docs so terrible that working off of SO examples really is the most efficient way to use it.)

Yeah, that's the kind of answer that you don't get anymore on SO, mostly because that kind of question isn't allowed anymore.

Nowadays questions must be much more specific, what has the consequence that the answers are much less generally useful.

There are less and less general questions because most of them have been answered already. If I see a question like that either I give a link to the docs or mark as duplicate. There's no point in writing such long answers if the information already exists somewhere else.

StackOverflow once tried to do an interesting community experiment around replicating documentation on everything but it lacked guidance / self-direction and they cancelled it.

I can't agree this more, once a moderator quoted about this in a moderation room: "I'm starting to become concerned that this room is somehow promoting the idea that all questions need to have an MCVE and/or that debugging questions are the only questions that are on-topic for Stack Overflow. That is not only wrong, it's horribly wrong, dangerously wrong. So I step in and correct this misconception as often as I can."

Some StackOverflow answers are so bad they are the worst resource on a topic.

Some are. That’s where you downvote or report it (depending on what’s ‘bad’)

Some of the bad ones are the most up voted one..

Things that technically work, but are not the best way, easiest or more secure way of doing what the person needed.

Often times when I find a solution to Stack Overflow the one I find most helpful is the 3rd or 4th most upvoted solution not the most popular one

>>> The US intelligence community actually keeps a clone of Stack Overflow

I feel this is a prudent idea. Not necessarily for a full blown start-up, but a community-funded resource. If you stripped out all the extras. And just cloned the questions and top 2 answers. The text based data set itself is very managable. Several gigabytes in scale. Not to mention its applicability for ML/NLP/codegen research ;)

Maybe you can use their dataset, https://www.kaggle.com/stackoverflow/stackoverflow

Source is here, https://archive.org/details/stackexchange

I think what the poster means is more than just unpacking the dataset. So the project should update itself every 12 hours, and has the capability of searching the information (using local database).

> the capability of searching the information

Not something SO is known for. Thus, it would be pure improvement over the status quo.

Hey when you mentioned US intelligence community do you mean the government agencies such as the army,cia or DoD

I’m just curious to know

The app Kiwix let’s you keep a static copy of stack overflow on your phone, it’s only a few gigabytes. It also has Wikipedia!

Dash on macOS also lets you download StackOverflow for offline use. It was my goto for working on airplanes, way back when traveling on airplanes was a thing.

Does it actually work? I tried a couple of times and the downloads never finished.

It worked for me when I tried it but it was so slow that unless you are running from an SSD it's not worth it (I had it on an external disk). Don't download everything, only get what you really need, the phone app was (maybe still is) a bit buggy.

Is there something like Dash/Kiwix for Windows?

Yes, Kiwix is available for Windows: https://www.kiwix.org/en/downloads/

There's also Zeal: https://zealdocs.org/

And DevDocs: https://devdocs.io/about

Thanks, somehow I managed to miss that for Kiwix.

The Q&A have a very permissive license, so anyone can try to host a static version. (This was on purpose, the idea is that people would have been less prone to answer, if the answers were trapped in SO.)

Moreover, there are a few spammer sites that already does this, and one of the problem of SO is to compete with them to be higher in the Google results.

So ... probably you will be able to read the Q&A in other sites, and after a while there will appear SO-likes site for niches.

In my opinion the usefulness of Stack Overflow is vastly overstated. It's nice to find a few examples to get started in a new framework that lacks documentation (when there's documentation at all...) but generally speaking non-trivial questions never get answered anyway. I haven't used it in at least 6 months, maybe even a year if my memory serves me well.

Stack overflow is very useful if someone already asked your question back when it was still accepting new questions.

That depends on your environment.

When programming in Haskell or Rust, going into SO is basically a waste of time, and the documentation or some specific search engine (yeah, hoogle) are the place to go.

When programming in C#, SO is basically irreplaceable, the knowledge there is extremely important and you just won't find it anywhere else.

When programming in Python or Javascript, it completely depends on what libraries and frameworks you are using.

> When programming in C#, SO is basically irreplaceable, the knowledge there is extremely important and you just won't find it anywhere else.

Not even in the official documentation? Wow.

The basic C# libraries are full of undocumented surprising behavior, and surprising behavior that is documented only on thematic docs and not on the function docs. And most of the ecosystem follow the convention.

There are many people that insist the .Net environment is well documented because there is documentation for every single item, but that documentation is far from complete.

My experiences with Stack Overflow until recently:

- Using it as a useful resource for small issues

- Seeing memes about how programmers essentially just copy/paste code from Stack Overflow all day long

- Reading comments about how Stack Overflow is severely outdated and any question will be shut down immediately

And then I had an issue with library A which was using library B but had a small inconsistency in how it used it that made my use case very difficult. So I posted a question. The developer of library B appeared, agreeing that it's a problem. Followed by the developer of library A, who agreed as well and said the fix would be in the next release. 2-3 days later, my bug was fixed by updating library A.

With a sample of one, that doesn't really say anything, my view of that website is a bit biased now.

I say this somewhat tongue-in-cheek, but I suspect it would probably result in more netsplits on freenode, owed to the inevitable influx of programmers who would now seek assistance on IRC like the good old days.

I've definitely noticed a downward trend of activity in the assorted programming chat channels I've frequented over the last 25 years or so, which I guess could be a result of the growing popularity of Q&A sites like stackoverflow.

But I still think it's one of the best places to seek assistance - plus, helping out programmers on IRC in real time is a good way to stay sharp and on top of the continuously evolving software landscape.

IRC is the first place I go if I can’t find an answer in the documentation or a search engine. If that doesn’t work, I’ll turn to a mailing list.

Personally, I find both of these options have less friction than starting a new question on SO.

100% agreed.

I also just simply enjoy lurking on IRC to assist other programmers in need. I learnt so much in those channels when first starting out, and these days I have a bit more spare time, so I enjoy "paying it forward" by helping others much like I was helped back in the day.

Plus there are a bunch of really nice programmers hanging out there, so there's a social aspect to it as well.

The key offering of stack overflow these days is less the ability to ask a stupid question and get told that you’re an idiot, and more the existing wealth of answers to questions that may be found by a search engine. I don’t think irc really produces answers which are easy to find.

I remember something Jeff talked about during the original Stack Overflow podcasts, that Google was the main interface for Stack Overflow. As in, your entry into Stack Overflow would for the majority of people be through search. And to speak for myself, this is how I use Stack Overflow 99% of the time.

This makes me wonder. Just having a copy of the SO data without the Google algorithms to bring you to the right question, the experience would probably be different when working of a copy.

I think it would have had more of an impact five years ago. Most of the best content is on blogs now not so. Most of the best so answers were really blog posts anyway

Finding more stuff on GitHub issues myself, but agree with the general sentiment that more content is elsewhere these days

Depends. General language questions are amazing on stack overflow. Once you starting getting into the quirks and bugs of individual frameworks, GitHub issues can start to be more helpful.

It would be nice if those blog posts were as easily aggregated as stack. I know many have tried to solve the problem, but it always feels like great luck when I find a good blog post

I wonder if we could implement a decentralized version of SO. It can be a CLI app since it is used by developers only. The app can keep all question/answers of topic relevant to the developer locally. This will allow developer to find answers offline as well. Since it is only text data it should not take more than a GB.

There's a lot of metadata not being caputred by MetaData which would be insanely useful for a lot of purposes. It would be kind of neat if a lot more of that was captured... and if SO answers had more programatic access.

The data is on archive.org and also on big query if you need it.

https://archive.org/details/stackexchange

with a human in the loop, this is IRC.

While SO is geared for developers. If someone takes this on, ideally it would work like StackExchange sites where one can exist for each topic.

- https://math.stackexchange.com/

- https://stats.stackexchange.com/

- https://3dprinting.stackexchange.com/

- https://ai.stackexchange.com/

- https://apple.stackexchange.com/

- https://serverfault.com/

- https://superuser.com/

- https://drupal.stackexchange.com

There’s a ton more of these.

Preferably a knowlage DB accessable through whois(1) protocol.

Whois does seem a little more flexible than dict. Able to delegate and recurse in answering a query. Let's add some flags

That, not more than a GB part is very wrong

    stackoverflow.com-Badges.7z (View Contents) 07-Dec-2020 17:09 273.8M
    stackoverflow.com-Comments.7z (View Contents) 08-Dec-2020 03:10 4.5G
    stackoverflow.com-PostHistory.7z (View Contents) 08-Dec-2020 22:50 28.0G
    stackoverflow.com-PostLinks.7z (View Contents) 07-Dec-2020 17:11 95.6M
    stackoverflow.com-Posts.7z (View Contents) 08-Dec-2020 18:48 15.9G
    stackoverflow.com-Tags.7z (View Contents) 07-Dec-2020 17:11 843.9K
    stackoverflow.com-Users.7z (View Contents) 07-Dec-2020 17:25 645.2M
    stackoverflow.com-Votes.7z (View Contents) 07-Dec-2020 22:14 1.2G

There's this open-source SO-esque Q&A forum: https://github.com/ASKBOT/askbot-devel

Oh we could start fresh with actual up to date answers!

You can add new answers to existing questions if the answer’s changed over time

I know, but most often the answer doesn't rise to the top and you have to look closely at the dates to have some context. Dead links and now-useless answers are my day to day stackoverflow experience.

If it’s that important to daily work it sounds like there’s a fundamental issues with documentation, training, and education in the world of software development.

Federated messaging seems popular these days. Maybe federated documentation is a good idea?

Perhaps git could be leveraged to distribute docs, examples, and questions?

Given how dated many of the answers are on Stack Overflow are, that refuse to get updated and prevent people from asking and answering similar questions on software versions that aren't from 2013, this has already happened but more slowly.

Before the World Wide Web and Stack Overflow, we used plenty of Reference Books and specific UseNet newsgroups.

Yes, it was a slower system, but it was just as good in the final result.

I think I use Stack Overflow once a month at the most, generally when I go there with a problem what happens is one of the following:

I have a problem I can't fix, I start writing down the problem, I realize as I am writing it down that it cannot be what I thought was the issue. I try other avenues for a few more hours and solve the problem.

I have a problem that I can't fix. I realize that I cannot reliably cut the problem down in minimal code to get a response on Stack Overflow because it is not a minimal problem. I try some other route to fix the problem.

I have a problem that as I am writing on StackOverflow I realize I have not tried a particular solution, I try it and solve my problem, I do not finish writing my problem.

I have a problem that seems like it is pretty esoteric, I ask a question, it never gets answered, I either manage to solve my problem, finding some github issue or similar suggesting the solution is a bunch of dependency updates or just not solvable. I either solve my issue or find some hacky workaround or do decide the requested functionality should be replaced by something else. No one ever answers my question, 6 months later someone says they have the same problem and did I solve it.

I have a problem that I actually finish writing and asking, someone responds and has misunderstood my question but their answer while wrong for what I'm answering gives me a clue on how to solve my problem.

I have something I want to do but I do not have workable code because I know absolutely no way about proceeding with the idea, so I do not ask StackOverflow because they just aren't helpful for anything that isn't simple.

I have a problem that is something like one of the previous situations, but I realize as I am writing it that it will get deleted or modded or something because it just won't work for StackOverflow guidelines, luckily most of the time when this happens I realize it should really be asked on one of the other Stack exchange communities. If I have progressed far enough to actually go ask a question on some other Stack Exchange community it tends to be answered reasonably quickly.

I have a problem in some area I do not know much about, perhaps because I am just started using the framework or something, my problem is simple but it does not work as I thought, I ask, it is a simple misunderstanding of how I thought it would work, someone responds with a solution really quickly and I fix my problem.

on edit: fixed bad grammatical error

on further edit: if StackOverflow went away pretty much nothing would change for me.

people who knows how to use web archives , and the ones who can search and click on show cached version on google search wont be much affected, i have developed a snippet management tool , so people can search there too , its https://codekeep.io

I wonder if the reason documentation is so crappy is because stack overflow exists?

Documentations explains internals usually.

StackOverflow benefits from also having answers to things trying to combine libraries and other things.

Funny enough, the inception of stack overflow was because another Q&A site called "Experts exchange" suddenly put up a paywall.

They had had a paywall for a while. They just had a bunch of bad answers and no way to judge whether the given answers were actually useful or not.

I don’t use stack overflow to much, I usually prefer documentation or blogs

I find that as I get more familiar with a stack and start solving more "complex" problems, then documentation and blogs are the way to go. But if I'm learning something new and need to change the background color if a button blue then usually stack overflow is the fastest place to figure that out.

programming quality would really improve.

People would have to git gud I guess.

$1 Billion damage I think is easy.

$10 Billion before it's re-created. Maybe. It'll take years to re-code and re-build the community.

$100 Billion. Doubt it. But not sure.

10 million visits a day, $1 per visit. Are we including all the Stacks? I think Server Fault could be much higher than $1.

https://stackexchange.com/sites?view=list#traffic

None. Most of the content on Stack Overflow is just people repeating what they heard from a more authoritative source looking for some internet points. (and often when it gets repeated it loses accuracy, context, and a real understanding of the issue at hand) There is a small subset of content that is original and not posted elsewhere but its not that common to be worried about. Often quality answers come from a person who has already published work somewhere else (eg. white paper, book, blog, documentation, etc).

That seems to me like saying that Google Search is insignificant because it only shows content from elsewhere.

Google isn’t repeating as us from memory where it could loose information/context, it’s quoting verbatim.

Also, Google can be annoying when it tries to be smart and the item you really need isn’t shown due to past behavior (the bubble it creates)

Maybe that should be shown as prominent/original rank to the query done. But whatever.

SO suffers from it's own version of: https://en.wikipedia.org/wiki/Chinese_whispers

There are a lot of answers to questions that authoritative sources don’t make the best job of explaining. Also, the more ways of explaining something the better in my opinion. We don’t all think on the same frequency and the more answers the more likely to find one on your own understanding. SO helped me many times when I got stuck on something. When learning something new I wouldn’t go directly there, Id use a tutorial or some authotitative reference

Completely disagree, and you're wrong.

Depending on the area you're looking into:

1. Many Stack Overflow answers do not exist in documentation at all.

2. Some answers are especially valuable because they touch issues in actual production (which are not foreseeable by documentation).

3. Answers (chronologically given) help ongoing document changes in APIs, which many documentation don't do or make it very difficult to do.

Take Python's Tkinter for instance - a lot of answers in in Stack Overflow supplement many things not covered by the official documentation. You'd either have to look at Tcl's documentation (but have no idea where to look, or how to read it), or figure out the source code yourself which will be confusing since the official documentation does not actually document every part of the library that it's sourced/ported from even though they exist for use.

4. SO answers often point you in the right direction, or to third party explanations and sources, and often compare between different semantic versions. (E.g. answers for D3 in JS often compare changes between APIs in the different versions 4, 5 and 6.) While official documentation sometimes chronicle their API changes, comparative usages/change in idiomatic usages are not always elucidated and it is in Stack Overflow that many of those are found.

I am wrong. For me using a dependency that can't be bothered to create proper documentation or uses SO as customer support is a non-starter.

If your attitude is that you "can't be bothered" to use a dependency when there is no proper documentation, you probably aren't working on difficult problems, or you're not really doing enough serious programming work where SO becomes the only resource available to solve a hard engineering problem.

Reply on news.ycombinator.com