Hacker News Reader

Comment by godelski | Hacker News Reader

I think this is mostly fair, but I think it could also be helpful to have AI generated "auto answers" that come with a warning. Like when you are typing in a question SE already tries to figure out if your question has been asked so there isn't a double posting. But what might also be nice is a stochastic answer that says something like "We have generated a possible answer, but be ware that this may not be accurate or to our community standards. If the answer is acceptable please accept and if it is not then reject and we will post your question." I'm sure someone could come up with a better phrasing. This would also de-incentivize people from posting AI generated answers and can make SE adapt to the changing environment rather than reject it outright (we do expect LLMs to get better, even if they are always stochastic parrots).

Would this not be a better "middle" ground? Thoughts?

View on HN

I think this could be dangerous because a generated answer might be 100% wrong and look correct, yet the question asker might accept it nonetheless. The more this happens, the worse the entire site becomes. I tend to agree with the concept of preventing as much AI as possible, simply because it is much harder to distinguish wrong AI answers than organic ones. Just my opinion though.

I’ve used plausible but wrong human code too. I understand the difference is scale but it’s worth remembering that people write answers like this too

Where scale really comes into play and gets scary is when bots can vote on other bots. Right now, most wrong human answers are pretty rapidly downvoted or corrected. But the supply of humans to make incorrect posts is limited and relatively balanced by the supply of humans to downvote them. Subtle errors in AI posts could become so widespread that it's impossible to counter them effectively.

They'd probably present it differently from human answers.

If people really want to be careless they can get AI generated code from copilot or chatgpt on their own already, I don't think this would be worse than that.

Just as someone who's helped people deal with nuanced reasons why exact SQL queries optimize better or worse under various scenarios, a hundred or more times on S.O., I think the reason they turn for human advice is that they place more value in an answer to their specific question that can't be generated by AI.

I've banned this account because of the Hitlerian (really?!*) URL in the profile. You can't propagate that stuff on HN. We'd ban an account for doing that in comments, and a profile is no different.

It's a pity, because you've also posted good comments and I think the proportion of good comments has been getting better over time, which is great, but that doesn't make things like the above ok. Also, you have a history of using this site for ideological battle and we don't want that here—it's not what this site is for, and destroys what it is for.

If you don't want to be banned, you're welcome to email hn@ycombinator.com and give us reason to believe that you'll follow the rules in the future. They're here: https://news.ycombinator.com/newsguidelines.html.

* Edit: on closer look, I can't tell if it might have been a bad joke instead.

You could just make the AI answer private to the asker. That way there's hopefully minimal harm to the repository of answers, it just stops containing questions that are easy enough for the AI to handle.

From the perspective of AI Safety (may get important soon), it is perhaps a very bad idea to allow AI agents to suggest execution/injection of arbitrary code into random places with a review from a single human that needed help from Stack Overflow.

[And don’t believe ChatGPT that claims it is an only a Language Model. It is not. It is a RL Agent trained with PPO.]

That's a fair point. Does the same argument apply to GitHub Copilot / ChatGPT?

Does it apply to things that aren't programming? e.g. People are already using these AIs for legal work.

One of the first steps of a runaway AI attempting to secure its presence would likely be inference/GPU/TPU compute access. And code injection is a vector. There are multiple other vectors.

But, perhaps considering AI as an adversary is a bad idea. And alignment along the “Love is all you need” lines is the actual solution. Tricky problem…

I was laughing last night when I realized that even if you want to make an AI that only knows and cares about maximizing paperclip output, you also need to teach it to love all humans unconditionally to prevent the apocalypse.

> You could just make the AI answer private to the asker.

So then the asker gets the wrong answer and no one else can correct it? Seems even worse.

That doesn't solve the problem that GPT-based AIs don't "know" when they're wrong. They will happily spit out authoritative-looking nonsense when presented with questions they don't know how to answer -- and the people asking the questions may not be able to recognize those answers as wrong.

I think this is a great idea, especially if the AI answer is tagged as such on the site. That way, future AI researchers will be able to avoid pulling those answers into their datasets and creating a feedback loop or whatever.

No middle ground. This is the wrong side of history.

If they stick to their guns, Stack Exchange won't be around in a decade. That's how fast the world is about to change.

Think about Copilot taking code from GitHub. If you start letting it contribute potentially shaky code back to GitHub, it's going to learn more incorrect patterns on its next harvest. It's a feedback loop. Anyone who wants accurate AI answers would not want the AI itself to contribute to the learning pool. The same is true for S.E. and ChatGPT. ChatGPT doesn't understand why something works or doesn't; it just infers the next word or sentence from the previous language. It's not factually accurate, but it draws from material that contains facts. If it's allowed to contribute back to its source data, it gets worse as a model.

I'm confident we're better than this.

Human evaluators can filter out the (for now) intermediate bad results. All of our AI models and products will climb a quality gradient year by year to the point where this will be increasingly less necessary.

To your other point, you can use machine data to bootstrap a "real" model. My team has done this to incredible effect.

You may be right in the long run, but it's definitely a gradient and the capacity for AI answers to be pasted thoughtlessly far outweighs the human capacity to correct it on a site that already creaks under the weight of insufficiently answered questions. I think if you ran such a site, you'd want to stay a little behind the curve of intermediate false but truthy-sounding responses.

Would it maybe help to extend https://developers.google.com/search/blog/2019/09/evolving-n... with new options, like rel="aigc" or rel="noTrainingOfYourLanguageModel" ?

> Think about Copilot taking code from GitHub. If you start letting it contribute potentially shaky code back to GitHub, it's going to learn more incorrect patterns on its next harvest. It's a feedback loop.

ChatGPT (and presumably soon CoPilot) learn via * Reinforcement Learning from Human Feedback*[1] on top of the raw language model. This takes low numbers of high quality examples to improve the quality, and avoids the "junk training data" entropy problem you identify.

The OpenAI Codex text-davinci-002 and text-davinci-003 models have been trained for code generation using this human feedback process[2].

[1] https://huggingface.co/blog/rlhf

[2] https://beta.openai.com/docs/model-index-for-researchers

>> ChatGPT (and presumably soon CoPilot) learn via * Reinforcement Learning from Human Feedback*

Technically, they aren't trained that way, their opposite recognition AI is trained that way. That's less than ideal when their original set of training data aren't drawn from their own output, only from a pool of humans influenced by their output. It becomes suicidal to the AI when it begins consuming its own output through what it considers to be trusted channels.

[edit] The ancient coder phrase "garbage in, garbage out" has never been more significant than now in the context of what's fed into neural nets and ML algos. I think it's basically silly hubris sprouting from a lack of underpinning technical understanding when people assert these models will train themselves without much more extreme means of assessing their output than can be provided by a few minimum wage workers somewhere.

> Technically, they aren't trained that way, their opposite recognition AI is trained that way.

No, they are trained that way. These aren't GANs or anything resembling that. There is a reinforcement model that is build from expert human guidance which is then used to fine-tune the LM.

From the HuggingFace explainer page I linked above: [there are] three core steps:

1. Pretraining a language model (LM),

2. gathering data and training a reward model, and

3. fine-tuning the LM with reinforcement learning.

Read https://arxiv.org/abs/2203.02155 if you want all the details (although the HF explainer is easier to follow). From the abstract of that paper:

> Starting with a set of labeler-written prompts and prompts submitted through the OpenAI API, we collect a dataset of labeler demonstrations of the desired model behavior, which we use to fine-tune GPT-3 using supervised learning. We then collect a dataset of rankings of model outputs, which we use to further fine-tune this supervised model using reinforcement learning from human feedback.

There is no "opposite recognition AI".

What makes you so sure of it? LLMs are just the novelty of the moment, no empirical evidence yet indicates that it's the direction the industry is going to take, it could just be just another fad like VR or web3.

Reply on news.ycombinator.com