Hacker News Reader

Comment by rightbyte | Hacker News Reader

Ye I signed up and tried some queries. It was quite scary.

"Write a function that sorts the elements of a c string backwards, in C." "Add a flag to the sort function which makes it sort the string forward." "Could you write a endian swap function for double argument in PIC assembler?" "Could you write a binary sort in PIC 8-bit assembler" "Write a Javascript function that prints the day of the week together with some Chinese words of wisdom."

It had no problem doing any one those. I ran them all, except the assembler ones.

The question is how good it is to process larger chunks of code and makes changes to it.

People thinking about becoming programmers might need to rethink their plans if this one improves ...

EDIT: Oh dear. I introduced bugs in its sort code and it found them and explained what they did.

View on HN

Last night I entered the page-long instructions for Advent of Code day 4, and it spewed out perfectly readable code and solved it on the first try [1]. And we're not talking about a common algorithm that has been solved many times before, but a convoluted story that is full of "elves cleaning up overlapping sections of the camp" (!), and ChatGPT was still able to understand it, write the code to solve it, and even explain how it works.

It's nothing short of a phenomenal milestone.

[1] https://twitter.com/GuiAmbros/status/1599282083838296064

Are you sure this whole problem with its solution wasn't indexed in ChatGPT training set, so all it did here is retrieve the whole answer to the problem similar to how google will find the solution pretty quickly if you search for "advent of code day 4 solution"?

Can you see if it solves an advent of code problem whose solution hasn't been posted yet? If it did I will consider this an almost publishable breakthrough.

It wasn't in the training set - the puzzle was released today, as part of this year's advent of code. ChatGPT is eminently publishable and has demonstrably produced plenty of novel, if boring, code.

Genuine question: can it write larger codebases yet such as: “write a pac-man clone in JavaScript, html, and css”

Yes, but you'll have to chop the project up into different parts and describe each sufficiently well.

For example, start with "write a level generator in JavaScript which outputs pacman like levels encoded in ASCII which could be used as the basis for a pacman clone."

Then you'd say "Great, now write a pacman clone in JavaScript which takes the input of the previously generated level generator and takes interactive input on the console."

Then you'd have to ask it to generate the HTML and CSS. It won't do it all at once. But it will do it piece by piece.

Yes, it's absolutely feasible. I did exactly that with a simpler version of this (Tic Tac Toe, and console only, no actual visual interface) but it worked pretty well.

The main thing is having a good idea of what the overall program structure will look like. If you have that, I think you can go pretty far with it.

Btw, in case anyone wants to see what working like this may look like, I recorded myself while doing that little experiment: https://youtu.be/TBMQIDMwI5E

It can't write pacman. But if you describe a scenario to it, it writes pretty accurate code and can drill down. Very impressive

I'm trying that right now. It can probably do it, the big issue right now is that it won't give really long answers and just stop in the middle, which can be avoided by asking him to finish it, but it doesn't always works.

If you can break down what you want into smaller functions, instead of just asking for the whole thing at once, you can actually do a lot with it, including telling it to out it all together at the end.

Won't link it again so as to not get too spammy, but I posted a recording of an experiment I did playing around with ChatGPT in that way on a sibling comment if you want to check that out.

I've tried giving it languages like Turbo Pascal or telling it to obscure frameworks like fp-ts and while sometimes it's not syntactically perfect, but I can guarntee it's not just scraping code snippets from somewhere.

The guy that got the first position for the first part, thus the first solution ever on that problem, was using ChatGPT to do it. [1]

[1] https://github.com/max-sixty/aoc-gpt

I tried higher level problems: write an http server in rust using warp and postgres, with endpoint for get users and put user.

9 compile errors, some due to cargo.toml issues (it only generated main.rs). But mostly worked.

The post we’re discussing is also pretty impressive, a working lexer/parser

> The question is how good it is to process larger chunks of code and makes changes to it.

> People thinking about becoming programmers might need to rethink their plans if this one improves ...

Very true, and this thought definitely crossed my mind as well. In the short term I imagine this is going to be like any other tool, it will increase your leverage, and make it possible for you to write code faster.

For example, I fed it 100 lines of a Django model and asked it to write some query code for me. Produced something that was possible faster than I could have. The few slight deficiencies it did have (optimizations) basically felt like me doing a code review.

The answer is it’s very bad at large amounts of code. It has no skill at reasoning across a large amount of files where say you’d need to make many small changes in a handful of places and know that those changes all work together.

These models are amazing at regurgitation. They have no skill at reasoning and any solution outside their training set. They have a distinct disability for precision in anything remotely novel or large. And they will bullshit subtle bugs all over the place at scale.

Even a junior programmer would produce more value than the next generations of this, simply because they can take vague English instructions and actually make iterative progress across a huge codebase, test those changes for all sorts of vague constraints, and be precise. There’s no ability for that here.

Well my mortgage is thankful for that.

I have a feeling though that tools like this will lower the skill threshold for programming by a lot.

I gave it this code I just made and asked it what it does and it figured it out:

    public class mean_sink {

        double sum;
        long n;
        
        public mean_sink(){
        }
        
        public void add(double d){
            sum += d;
            n++;
        }
        
        public double getMean(){
            if(n == 0)
                return Double.NaN;
            return sum/n;
        }
    }

I asked it to improve the numerical accuracy and it did it with BigDecimal instead. I asked it to only use doubles and it did some strange reciprocal summation when I was "hoping" for Kahan summation algorithm. Its "reciprocal summation" did not work. But it made a correct Kahan when I asked for it.

I dunno. This is insane anyway. It is really hard to grasp the consequences.

This tool would make for a great pair programmer and an intelligent interactive rubber duck

Except you must have the underlying knowledge to effectively review code. So how that would obviate a career in programming is a strange choice by the grandparent commenter. Anyone can rubber stamp bad code, but to stop bad code from entering projects you need experts.

Most projects have plenty of bad code, nobody cares, as long as it solves the problem.

The bad code it produces is not just stylistically bad, it often doesn't solve the problem.

It seems to be making the same kind of errors in code as we've seen in speech, you get a lot of plausible but subtly wrong hallucinated content that only someone who knows the subject can identify. It prefers probable output over matching the specifications so it'll add unnecessary branches just because it's seen them millions of times.

Among errors I've seen it make there is bad indexing, adding conditions that serve no purpose or assuming a variable has a specific structure when it doesn't.

>Even a junior programmer would produce more value than the next generations of this

At the rate this is evolving that wont be a limitation for long.

My 14 yr old son thats taught himself to code in c# over the last few years is going to be put of a potential job before he cab be taught to code professionally.

This is quite frightening, the rate of evolution.

I already managed to get it to write an implementation of Pong and a Chip8 interpreter, both in C and using SDL2 for graphics/input/sound (it even wrote a function to generate the waveform for a beep for Chip8 and rebound sound for Pong!). I'm pretty sure I can get it to generate all the code for a Gameboy or NES emulator, I've taken a few tentative steps and it definitely can reproduce the memory map and generate code to parse iNES/GB cartridge headers

Something I’ve noticed in my own experimentations is that it’s really only limited by the length of its replies. It can happily generate perfect code but then stop in the middle of a function simply because the website only asked for 200 tokens or something. It’s really quite something to say that one of my gripes lies not in the model but in the user interface wrapping the model.

Hi, thanks for breaking it down.

Do you write code close to the machine at $job?

I wanna try with more feature and UI heavy request.

I actually comment because of your edit in your response : you manually introduced bugs, reran the process, and it seen you broke it ( fine ) but was able to explain it to you in English ?

Where is that comming from ? Adjacent comments and git comments?

> I actually comment because of your edit in your response : you manually introduced bugs, reran the process, and it seen you broke it ( fine ) but was able to explain it to you in English ?

I fed it it's own sorting function but I added bugs to it and asked it to spot the bugs.

I asked it to make some stuff in flutter and it worked well. I did notice sometimes it just adds a comment in the function saying “insert logic here for X” when it doesn’t know how to write it.

I tried a hard level leetcode problem on it. Perfect results on the first attempt.

it was trained with leetcode data though?

Ah I didn't know that.

programmers are final worker. there are not no more jobs.

Reply on news.ycombinator.com