Hacker News Reader

Comment by can16358p | Hacker News Reader

I'm a long time StackOverflow user.

A few days ago I had some confusion about how to use a React Native library and docs were also not super-clear for me.

Asked StackOverflow. Then I said "let's see what happens if I copy paste the exact same question into GPT-4", and did it.

Had my answer in 30 seconds, which explained the issue very well and worked. A few hours later I got an answer on SO too telling almost exactly the same, which was right.

Anectodal for sure, but ChatGPT and similar tools will keep evolving faster and will become only better.

Not surprising to see decline in SO. But I believe there will be an equilibrium as LLMs will need that training data too.

View on HN

Without the answers from SO, ChatGPT quality will decline as well. Wait 2-4 years for new tech come up, ChatGPT won't have the training data and noone is answering on SO anymore. You would just get wrong answers.

The ChatGPT of 4 years from now will get its answers from the copilot-enabled editors of people who know how to do the thing. If we're lucky it'll also be paying those people royalties, but I'm not holding my breath.

How does it distinguish between people who know how to do the thing and people who use copilot, but are terrible at their jobs?

It gives bad advice to users, from which it learns that that advice didn't work, and only the good stuff gets used going forward (may take some time to retrain and incorporate user feedback). I expect a proliferation of advice that works well enough to convince you that your problem is solved, but which might be subtly broken in ways that you have to come back to.

Tracing the fix-it sessions backwards in time to the broke-it sessions (such that the AI can learn from its mistakes) might be tricky unless we somehow manage to connect copiloting sessions through git history. Imagine a big database of changes where the AI is waiting to see whether the code persists, or if somebody comes along and fixes it.

vscode is owned by microsoft and could send runtime results back.

so is github...

ChatGPT can evolve. If it doesn't know the answer you could be identified as an "expert" in the field and you could be asked for an answer. This is how ChatGPT will help connect humanity with all our experts. Basically we'll all be training the model and getting information from the model. We're going to have to get serious about thinking through oversight so bad actors can't corrupt valid information.

ChatGPT can answer the homework and lazy developer questions that comprise 99% of SO, so maybe SO will just have the hard & interesting questions.

One can hope, anyways.

LLMs can navigate sources and generate both documentation and potential answers directly from sources.

Many questions on SO are exactly about things that are not covered by the docs of the tool/language/library in question.

This is wrong especially for programming where a program can execute a program. GPT-x will learn by itself from programming. No data training will be needed.

This works for now, but I wonder what will happen for brand new Framework X in 3 years time if people have stopped contributing to sites like SO. Where will ChatGPT get the knowledge to process and summarise and feedback to people? Blog pages?

As I understand it, someone needs to be creating the information that goes into the top of the funnel. I don't think ChatGPT can create an answer from scratch.

That's exactly why I wrote the last part.

Ok, but where is ChatGPT getting that answer?

I think this is a major problem with LLMs. You often want to know where the answer came from not just so that you can verify the correctness, but also so you can do your own research, learn adjacent topics, and solidify your knowledge.

It seems like there's an opportunity here for SO to use an LLM to provide an immediate response to questions, then allow the community to discuss, clarify, expand, and even correct the response.

This could even be done collaboratively with the LLM to improve its training with real expertise instead of feeding back into itself.

This would be a bit of a product shift for SO though, and some people would undoubtedly be opposed to it and leave the platform. But if LLMs pose a threat to the platform it may be better than to alternative.

Considering SO's corpus is liberally licensed, couldn't anyone take up this project?

Had my answer in 30 seconds, which explained the issue very well and worked. A few hours later I got an answer on SO too telling almost exactly the same, which was right.

Wow, how strange...

If you imply the person copy pasted it from ChatGPT, it's probably not as the answer was from a very reputable user especially on that framework, and even if the essence was the same as the GPT answer, it wasn't in the same style.

I've been learning to code for the past year. StackOverflow was hugely important in the beginning. I haven't used it in probably 2 months because ChatGPT does an excellent job of not only answering my questions, but also allowing me to ask follow up questions like "what does that specific line of code do?"

I’m most surprised with you getting an answer from SO, instead of getting your question shot down by overzealous mods

Reply on news.ycombinator.com