Comment by jchw | Hacker News Reader

Comment by jchw | Hacker News Reader

Parent Full thread

Not very impressed with either Copilot or ChatGPT for programming purposes. I'm sure it will be interesting for some people, but whenever I really feel like I could benefit from an intelligent AI, it usually falls flat. Most commonly, it will just generate calls that don't exist, and if you try to get it to implement them, it will write bullshit. Now the thing is, maybe it really does improve developer productivity by virtue of letting the developer autocomplete boring things. To be fair, I did see some opportunity here. But on the other hand, it also seemed to harm productivity, as I had to carefully review everything it did, and eventually had to disable Copilot so that it would stop annoying me with nonsense completions every time I stopped typing. I'm not really sure if language models are actually getting closer to what we really want, or if they're getting closer to some other maxima that just looks similar. Clearly they're doing something that is novel and probably even useful, but with increasing parameter counts it seems that the underlying problems are not changing. Something certainly seems like it must be fundamentally wrong...

I use them all the time with success. For copilot it’s very useful for writing tests and autocompleting repetitive code. Like writing a Typescript interface that matches a JSON object, or listing properties of an object into an array. ChatGPT is at worst a good rubber duck, and at best a superpowered documentation tool. If I ever get confused by docker or some popular JavaScript library it’s able to give me a custom tailored answer that usually solves my problem on the first or second try.

Neither of them are any good at just writing a ton of code (unless the code is mostly boilerplate). But they’re amazing keystroke savers and a next generation stack overflow.

How often are you seeing bogus documentation? I see that concern often mentioned. I tried ChatGPT and the first question got an obviously wrong answer. That seems worse than stackoverflow to me. Maybe I'm not holding it right?

I think it entirely depends on how popular the library is. React for example is so common online that GPT has the whole API figured out.

Even if it’s wrong the stakes are low. Just paste the error back into the chat and it will give you an alternative.

But the real solution here is the new Copilot X for docs. Feed in the code base and all documentation and now you have the real source of truth at the helm.

What would be amazing for Copilot + Typescript is if it could somehow integrate with the TS compiler (or perhaps just get sufficient fine-tuning on the TS type system) so that it only generates valid/type-safe suggestions.

I think about this often. If copilot could hook into the Typescript language server, it would be much less likely to hallucinate and generate blatantly wrong suggestions.

I second this. My impressions so far are pretty much the same. Have been trying to get ChatGPT to generate code for some algorithms that are pretty close to well known implementations but with some twists and it absolutely fails no matter how much I tried to provide hints to change the suggestions it gave me. In the end I just went back to hand coding them since it actually saves me the frustration of trying to get ChatGPT to bend in the direction I need.

Have you tried GPT-4?

Will give it a go. But probably it’ll take another generation or two before it’s good enough.

Cool. I’d be interested in the algorithm that you’re giving it to write and what problem it is having with the output.

ChatGPT << GPT4 when it comes to coding.

Reply on news.ycombinator.com