It reminds me of working with a junior developer, you need to understand how to communicate with them effectively.
Having used it for the last week or so to help me translate Python -> Rust, I'd say GPT4 is on par with even senior engineers for anything built in. The key areas it falls down on are anything involving org/repo knowledge or open source where things can be out of date.
There's a few teams working on intelligent org/repo/opensource context fetching, which I think will lead to another 10X improvement.
There’s still a lot of need for a human to be involved, buts it’s not hard to see how this will make individual developers more productive, especially as LLMs mature, plug themselves into interpreters, and become more capable.
Not to mention that humans will stop learning how to program, since they won't need to as the AIs will be doing all the coding for them.
There'll be a general de-skilling of humans, as they offload the doing of virtually everything to robots and AIs.
Then Idiocracy will have truly arrived, only it'll be even worse.
You thought GUIs were bloated and slow before? Just you wait!
https://gist.github.com/int19h/4f5b98bcb9fab124d308efc19e530...
Take a close look at the SQL queries it generated to answer the question. On one hand, it's very impressive that it can write a complex query like this as part of its reasoning in response to a question like "pick a bride for your son":
SELECT name, age, clan_name FROM (SELECT * FROM people WHERE name IN ("Mesui", "Abagai", "Yana", "Korte", "Bolat", "Chagun", "Alijin", "Unagen", "Sechen")) AS potential_brides INNER JOIN clans ON potential_brides.clan_name = clans.name WHERE clans.tier = (SELECT MAX(tier) FROM clans WHERE kingdom_name = (SELECT kingdom_name FROM clans WHERE name="Oburit"))
So, yes, I think you're right. The more code out there will be written by AI, the more it will look like this - working, but suboptimal and harder to debug.
I think with LLMs we will start to see code with better and more descriptive comments, since there is significant advantage to be found when the code has as much information and context as possible.
Human programmers often hoard that for job security or out of laziness.
I agree, I avoided saying that in my original comment to avoid nit picking. The depth of knowledge is incredible. It’ll just make junior level mistakes while writing senior level code.
> The key areas it falls down on are anything involving org/repo knowledge or open source where things can be out of date.
With GPT-4, I’ll paste snippets of code after the prompt so it has that context or even documentation of the API I need it to work with. The web client is still limited in the amount of tokens it’ll accept in the prompt, but that is expected to increase at some point
More intelligently putting it in a loop & other helpers for it (which we're now seeing worked on more) should massively improve performance.
That has taught me some things I didn’t know and helped me formalize and understand concepts better (I think).
Also, on a more funny note, it’s the best ffmpeg interface I’ve ever used :)
I asked it to help me extract certain values from a spreadsheet. It gave me a complicated and overly long formula using FIND and MID that didn't even work.
GPT-3.5, otoh, gave me a neat little REGEXEXTRACT formula that worked perfectly.
When I pointed out to GPT-4 that it can just use regex instead, it "apologized" and again rewrote a formula that didn't work.
Which- maybe that's just the thing, maybe that's what people are using this for right now
(This could change once it gets to the point of stamping out entire projects, but for now)
As far as debugging, I find that I can just say "are there any bugs in that code?" and it does a great job at finding them if they're there (even though it's what it just gave me)
The first one was to write a python script to watch a list of files given as cmd line arguments and plot their output anytime one of the files changed. It wrote a 100 line python script with nice argument parsing to include several options (like title and axes labels). It has one tiny bug in it that took a couple of minutes to fix. When I pointed out the bug it was able to fix it itself (had to do with comparing relative to abs file paths). If I wrote the script myself I would not have made something general purpose and it would have taken maybe 30 minutes to do.
The second example required no fixing and appears bug free. I asked it to write a python function to take a time trace, calculate and plot the FFT, then apply FFT low pass filtering, and then also plot the filtered time signal vs the original. This is all straight forward numpy code but I don't work with FFTs often and would have had to lookup a bunch of different API docs. Way faster.
I have also had it write some c macros for me, since complex C macros can be hard to escape properly and I'm not comfortable with the syntax. Its 100% successful there.
This is the thing: you don't need to be exact. Even very vague gets a long way.
you can write prompt to clean unstructured text into structured text. that becomes the work area.
instead of asking gpt "write me some code in pyton that does x" you can just tell gtp "do x with the structured data in the work area and present a structured result"
gpt becomes the computation engine
granted it's super expensive and quite slow, today
but with a few plugins to take care of mechanical things and the token memory space becoming increasingly bigger, writing the programming step directly on there may be the standard. as soon as vector storage and reasoning steps gets well integrated, it's going explode.
One thing I could see is a "self updating" script, that adjusts its logic on the fly when new data are found.
I'm a member of the management team for a company, and everyone was very excited for ChatGPT. As someone with a technical background, the caveats were a lot more obvious to me from the start. Treating it as an incredibly fast, eager junior developer who is desperate to please and submits code without testing it has made it a lot more usable.
zipinfo -l foo.ipa "Symbols/*" | awk '{t+=$6}END{print t/1024/1024}'
It took a lot of prompting to guide it to the correct answer:
The weird part to me is how sure it is of itself even when it's completely wrong. (Not too different from humans I work with, self included, but still...) e.g.
> "The reason I moved the subdir filtering to 'awk' is that the 'zipinfo' command doesn't support filtering by subdirectory like the 'unzip' command does."
Where does it come up with that assertion and why is it so sure? Then you just tell it it's wrong and it apologizes and moves on.
Now, the addendum here is that if either of us had read the zipinfo man page more closely, we would've realized it's even simpler:
zipinfo -lt foo.ipa "Symbols/*" | tail -1
I also tried to use it to translate about 10 lines of Ruby that was just mapping a bit of json from one shape to another to jq. It really went off into the weeds on that one and initially told me it was impossible because the Ruby was opening a file and jq has no I/O. So I told it to ignore that part and assume stdin and stdout and it got closer, but still came up with something pretty weird. The core of the jq was pretty close, but for example, it did this to iterate over an object's values:
to_entries | .value.foo
[].foo
One of my pet peeves is developers who never bother to learn idiomatic code for a particular language, and both of these turn out to be that sort of thing. I guess it's no worse than folks who copy/paste from StackOverflow, but I kind of hoped ChatGPT would generate things that are closer to idiomatic and more based on reference documentation but I haven't seen that to be the case yet.
---
Has anyone here tried using it for code reviews instead of writing code in the first place? How does it do with that?
It's always "Sure thing!" or "Yeah I can do that!", and it responds so quickly that you equate it to when a human responds instantly with a strong affirmative - you assume they're really confident and you're getting the right result.
If it took 5-10s on each query, had no positive framing and provided some sort of confidence score for each answer, I think we'd have different expectations of what it can do.
As for ChatGPT i have jokingly thought to always tell ChatGPT “that didn’t work” until it settles on something it is really confident on (no more alternatives) then try that.
Unfortunately the gpt api is non deterministic, probably due to stupid censorship requirements.