Hacker News Reader

Comment by throwaway09223 | Hacker News Reader

You have to put it in context: How good is the human alternative?

I have interviewed hundreds of engineers across the entire skill spectrum. I think GPT4 right now is about on-par with a mid level developer.

It makes a lot of mistakes, especially around counting, and usually can notice and fix them if they're pointed out. Humans do this all the time -- how often do you get a compiler error from something silly?

It often misapplies interfaces on the first go-around. Again, this is just like humans. We make mistakes, notice them, fix them. If you simulate this by telling GPT that its code produced an error it will often correct itself.

The absolutely killer feature of GPT4 is that it has these skills in every subject. It's fluent in kernel operations. Databases. Networking. Various UX frameworks. Any language.

It's definitely not perfect. But, if the alternative is hiring a mid-level human engineer, GPT4 is a really compelling alternative.

View on HN

A mid-level human engineer can iteratively fix the mistakes they start making, instead of that just being the final result. Can GPT-4 do that?

Yes, like I said if you give it the results, eg "That code resulted in this error. What's wrong?" it will more often than not correct its mistake.

Hooking it up to automatically run the code in question and examine the output is a trivial undertaking - many folks have already done this.

My experience has usually been that it will apologize and then produce another snippet with a different flaw. When that is pointed out, it will usually go back to the original snippet with the original flaw. It sometimes also insists that it has “run the test suite and they all pass”, just like a human programmer that is trying to fake it until they make it I guess?

I wonder if anyone has done a feedback loop with GPT and compiler errors to train this sort of thing

they have, with langchain

It does it pretty well for Python with code interpreter, especially for quant/ ML heavy code.

> I think GPT4 right now is about on-par with a mid level developer.

At writing code, that might be a slight exaggeration. At explaining code it's at least at mid level, possibly better.

How many mid level developers do you know that understand the syntax and apis of every single programming language in existence?

I’ve used chatgpt recently to build an entire c++ game with sound, keyboard and mouse control, ai, etc, having never used c++ before. I’ve also never built a game before. I’ve worked with a lot mid level developers and I’ve never seen any that can write me a a c++ file giving me working code for a collision system in 30 seconds. Let me know if you find one though.

I would peg gpt as between junior and mid. Mids (around 5-10 years of experience) tend to produce overly complex code for the problem at hand.

Dude, chatgpt as a junior or mid level programmer? Have you actually met junior or mid level programmers?

I’ve got 20 years of experience in the tech industry and chatgpt is far far better than any mid level engineer I’ve ever met and I include FAANG engineers too.

Chatgpt can answer any leetcode programming question in essentially every programming language in existence in less than 20 seconds.

Chatgpt can analyze classes for errors written in every language in seconds.

Chatgpt helped me write animation, ai, sound handling code in c++ and helped me build a full working game, the total code generation time for chatgpt “thinking” was on the order of one hour.

I don’t know any senior engineer who could help me (a c++ newbie) write a whole working game engine with all the necessary systems in less than one hour of “thinking time”.

I’m starting to think that all the “naysayers” of chatgpt simply used it once or twice, saw one error and dismissed the whole thing. Or they’re simply afraid for their own skills and avoid chatgpt as the alternative scares them too much.

Can you share a link to the game you made with ChatGPT?

Yes, we employ them regularly. And yes, I use both ChatGPT and copilot regularly.

In my experience, if you compare the unedited output of ChatGPT to the unedited output of someone with a couple years of experience in the given domain, ChatGPT will have more subtle bugs than the human provided output.

Hence my between junior & mid assessment. It doesn't mean that it's useless though. Having a tool at that level of experience for every domain in the world that can cook up code near instantaneously is damn useful. And I assume it'll only get better.

Reply on news.ycombinator.com