Hacker News Reader

Comment by chatmasta | Hacker News Reader

My experience using GPT-4 for coding is that it's got the knowledge and skill of a senior engineer, with the high maintenance of a junior engineer. You can get it to output small sections of quality code, but the amount of prodding it takes to piece it all together means you may as well have spent the time writing it yourself. But the future of GPT as a coding assistant is definitely bright. It just needs more chaining, so I feel less like its assistant, asking it to come up with prompts and then pasting them back to it after iterating on the code.

View on HN

Those aren't very good senior engineers then lol. GPT-4 has the coding skills of an overconfident junior engineer and little else.

I agree the future of GPT-X models as coding tools is bright. But for actually doing engineering work outside coding (or even delicate changes in an existing code base) much less so.

I'd love it if junior engineers organized code as well as ChatGPT. I'd also love it if all engineers stopped mixing spaces and tabs.

Add a linting step to your CI to catch this stuff?

Maybe I could get chatgpt to write those pipeline steps for me. :)

You can and you should. It’s this kind of busywork I farm out to GPT these days and it’s great at it. I suck at it, so it saves me an hour to focus on what I’m good at!

That can mostly be fixed with conventional prettify tools. No modern ai required

Do you have a style guide?

ChatGPT, on spaces vs tabs: https://chat.openai.com/share/b2b0be49-54f9-4f73-a753-3edbd1...

Well GPT 4 consistently defaults to returning 2 space indents, so clearly it has an opinion on the matter.

Of course you can't really compare it to a real senior engineer, but it has a lot of traits that come close or even surpass seniors:

1. It picks very good variable names

2. Clean, nicely structured code

3. Vast amount of knowledge where even a senior engineer still needs to look up stuff (I don't know about you, but I have a terrible memory. ChatGPT clearly has excellent memory about API's, regexes, etc.). It's the "I don't even have to look at the docs or StackOverflow for this" type of knowledge.

4. It's fast, like very fast, like "I don't even have to think about it and just type it out perfectly like a maniac".

GPT-4 has incredible breadth and no depth.

It's an excellent complement to a strong programmer, who will have incredible depth -- and may have breadth relative to other coders, but will be very narrow in the whole space of programming.

Just don't use it for things you're already an expert on, except perhaps for starting out / bypassing boilerplate.

It has depth as well. You just need to copy in the relevant docs.

It’s writing fault tolerant data scraping scripts for me I don’t know how to write myself.

The main thing it's lacking is a client-side merge/error-detection engine which combines all of the previous outputs.

Anything over ~100LoC and it gets sloppy including it all in the code block. Which is fine, because I don't need it in your context window if that bit is working and you're not relying on it.

Though I have gotten outputs up to nearly 250 lines from it..

Automatically feeding back code details + Traceback messages also fixes errors decently often enough.

Agree. However I noticed an unintended benefit of using GPT for coding is that it helps you think carefully about the problem you are trying to solve when writing the prompt.

"Language as a tool of thought" is a technology we often neglect we are equipped with.

This is essentially what rubber ducking is.

That is why most developer jobs are safe. Tasks from stakeholders often contains only a title.

i don't use it that much for code but when i do it's for a specific function or method. I describe the inputs, the logic i want, and the outputs i need. So basically i have it worked out in my head I just let chatgpt type it out for me. Any mistakes it makes are pretty easy to catch when using it this way.

Rubber ducking in the age of WFH is definitely a use I've found for it.

It’s still way faster than I could hope to be. It’s a true refactoring machine with 0 mental effort. “Here is this code block, make it do this other thing, add this feature and argument and error condition” etc. It augments the amount of output that’s possible to achieve at any seniority level.

> so I feel less like its assistant

Ah so I'm not the only one who feels this way lol. It's as brilliant as it is dumb.

It's Einstein with dementia/alzheimers.

They just put out a blog post about how they’re doing exactly that by rewarding chain of thought style reasoning.

Generative AI will not be able to do that in any true sense. Not the way you're thinking about it.

For me it is not even the code per se but how it helps me with naming database tables and properties when I describe the broader scope.

The future is more than bright, especially with a new model that is more current.

Offtopic but just today I was aksing it for the most current versions it knows of right now:

Python: 3.9

JavaScript: ECMAScript 2021 (ES12)

Java: 16

C++: C++20

C#: 9.0 (.NET 5.0)

Ruby: 3.0.0

Swift: 5.4

Go: 1.16

Rust: 1.51.0

TypeScript: 4.3

PHP: 8.0

Kotlin: 1.5.0

Scala: 3.0

R: 4.0.5

Perl: 5.32

And frameworks:

Python:

Django: Version 3.2

Flask: Version 2.0.1

Pyramid: Version 2.0

TensorFlow: Version 2.6.0

PyTorch: Version 1.9.0

JavaScript:

Node.js: Version 16.9.1

Express.js: Version 4.17.1

React.js: Version 17.0.2

Angular: Version 12.2.4

Vue.js: Version 3.2.6

Java:

Spring Framework: Version 5.3.9

Hibernate: Version 5.4.32.Final

Struts: Version 2.5.26

PHP:

Laravel: Version 8.54.0

Symfony: Version 5.3.6

CodeIgniter: Version 4.1.4

Ruby:

Ruby on Rails: Version 6.1.4

Sinatra: Version 2.1.0

C#:

.NET Core: Version 5.0

ASP.NET: Version 5.0

Entity Framework Core: Version 5.0

Oh that's interesting, I didn't think about asking it directly about what version it thinks is current. I wonder if it can hallucinate here?

It's really irksome when it tries to use functions which were renamed or removed. This could be detected automatically (and possibly remapped in some cases)

I got mostly the same versions as you, both on chatgpt3/4, using english.

If you ask it in other languages the minor version also seems to change often.

Finnish gives you dates, and code-davinici-edit-001 is 1 major release behind on almost everything.

Tässä on luettelo ohjelmistojen viimeisimmistä vakaiden versioista:

Python: 3.9.5 (30.4.2021) JavaScript: ECMAScript 2021 (23.3.2021) ECMAScript: ECMAScript 2021 (23.3.2021) Java: JDK 17 (28.9.2021) C++: C++20 (20.2.2020) C#: .NET 6 (8.11.2022) Ruby: 3.0.2 (24.8.2021) Swift: Swift 5.5 (20.9.2021) Go: 1.17 (16.8.2021) Rust: 1.54.0 (27.5.2021) TypeScript: 4.4 (28.7.2021) PHP: 8.0.9 (29.7.2021) Kotlin: 1.5.31 (26.8.2021) Scala: 2.13.6 (17.2.2021) R: 4.1.0 (18.5.2021) Perl: 5.34.0 (30.5.2021) ... Sinatra: 2.1.0 (10.4.2021) .NET Core: 6.0 (8.11.2022) ASP.NET: 5.0.10 (19.8.2021)

All it is doing is responding with things that look like accurate languages and version numbers. They may be accurate but are absolutely not representative of what it has been trained against.

Asking it directly is a good way to get a response containing version numbers that look correct, but not a good way to actually indicate which versions it's been trained on.

A better method would be to look at the language features it's using and infer from there. Or, better still, look at which versions were out when the training data was collected (which I believe is September 2021, don't quote me on that).

Is there a risk of it hallucinating what new features could do instead of basing information on having seen them before?

Yes, good point, I suppose we would have to weigh the probability of that against it fudging a number. I would assume inventing a new feature is harder, but who's to say; all this LLM stuff comes down to probability.

i just asked it "what's the output of python -v" and the little sample code reported python 3.9.2.

I got:

    Python 3.8.5 (default, Jan 27 2021, 15:41:15)
    [GCC 9.3.0] on linux
    Type "help", "copyright", "credits" or "license" for more information.

And a lot of junk text.

Ha! So far it seems four posters here have asked similar questions and got four different python versions.

I don't know if we can treat the gpts in this way and expect reliable answers. We just get pretty good answers right up to the point that we don't.

Agreed, I think this is where its "stochastic parrot" nature shines through. All those version numbers are probably equally well represented in internet text.

It does not actually mean much. These chatbots can easily output JavaScript using var where it absolutely should not. It being aware of the latest standards does not mean it can properly use the new syntax or features -- it emits a form of whatever crappy/legacy code it was trained on.

But this is just a plausible looking answer, it is still a text generator, this has no bearing on reality per se.

Reply on news.ycombinator.com