Stack Overflow bans ChatGPT as 'substantially harmful'

I think the problem is that people cannot envision (or cannot believe, when told) that something that can speak human language would outright bullshit its way through every single conversation.

And because GPTs and chatbots are built as very convincing bullshit generators, some percent of the time (majority of the time) they will say things that check out - and that provides enough fuel to start building false assumptions that everything else it says is true.

On the bright side, dumb people have always been around. It's not like it's getting worse, in fact I would bet it's getting better. The meta-phenomenon is not something new here.

That people ask it programming questions or they use it to poison and spam online communities for internet points?

They didn't care about the harm, they just needed an excuse to close source all their models.

I have no idea.

This thing allows to produce those horrible or incorrect answers order of magnitude easier.

I am big fan of Gpt and not a fan of restricting its usage, but i think this decision is completely correct.

Considering that, stack overflow is definitely one of the source materials for Gpt training, and probably a very significant one for like Codex.

If i ask on SO it’s because i want a human answer. What is the point of getting a Gpt one?

If I wanted the Got opinion, I can __just ask gpt__ (and btw I already do it always, now).

So the decision is very reasonable.. If anything, it’s sad that i cannot see any way in which it can actually be enforced

Until it stops bullshitting, absolutely not.

I've asked it to write me some Python code for the Redlock pattern [1] and whilst it generated text that explained it correctly the code wouldn't run - it assumed the class it needed was called `RedLock` when instead it is simply `Lock`.

I've also asked it how you can make a Flask application not write any cookies on a specific route. First it told me I can use the `Flask-NoCookies` package, gave me the `pip install flask-nocookies` command and showed me how to use its decorator. Great! Except there's no such package called `Flask-NoCookies`. It's just made it up.

I've just repeated that question now and it's told me I "can use the `@app.route()` decorator and specify the `no_cookie` option", again, this doesn't exist and it's just made it up.

There are also examples people have posted to Twitter where it's returned references to books, papers and URLs that it's entirely fabricated, too.

1 - https://redis.com/redis-best-practices/communication-pattern...

The article already states why this is such an issue: The results could be correct or incorrect (the model has no idea either way), but are delivered in the style of a well-researched answer. They can also be automated, so there's tons of these deceptive answers.

I don't see it this way; Without curated information we wouldn't have a resource of valuable Q and A's for AI to base itself off of. I'm ignorant on what particular text the model was trained on but I have to assume it had a lot of details from online resources. It knows about coding languages and common problems so it has trained it's information form something. Without stack overflow or some kind of relationship between openAI ChatGPT suggested answers and confirmation that it was valid the information would becomes pointless for future frameworks since everyone would be stuck using ChatGPT for answers and it would have no place to find the answers. Maybe I'm missing something a context of some kind but I don't see it providing valid answers without the wealth knowledge like a place that stackoverflow can provide.

I don't think that you have the data to back this up.

I have trialled live coding with it and while amazing it suddenly generates half outputs and gets stuck. Right no, no, its swamping their volunteers, not suddenly making their service better.

There's a big concern in the long run: if ChatGPT seriously reduces the number of people who visit StackExchange domains, there will be no dataset no for GPT 4 / ChatGPT 2. E.g. what if a brand new programming language gains popularity, or new libraries with very different patterns of use?

This "paradox of reuse" is a really big deal, IMO (blog post on the topic: https://nmvg.mataroa.blog/blog/the-paradox-of-reuse-language...)

It's actually a separate concern from the spam / content bloat described in the linked post, but they complement each other in creating amplified harm for StackExchange-like platforms: some fraction of users may stop visiting the site (because ChatGPT answered without links) and some other fraction of users will submit spam to try and earn points.