Hacker News Reader

Comment by izzygonzalez | Hacker News Reader

I’m not sure if the rest of the responses here are reflexive self-soothing, or just caught on the “ChatGPT” product itself, but unequivocally, your anxiety is warranted.

Code generation is moving extremely fast. This tech didn’t freeze in time at Codex or Copilot or ChatGPT. It’s one of the most exciting and difficult domains in AI and the smartest people are all set on solving it.

I’m sorry you’re feeling distress. You’re in good company. A lot of the world is going to have to deal with these problems very soon.

View on HN

I just disagree, no one with any level of software accumen think AI generated code can write anything beyond a high school student copy and pasting.

The vast majority of software engineers are hired to work on is deeply complex in ways humans barely understand. It isn't small little JavaScript loops, it's huge multilayered executables with an incredible number of interdependences. It takes 6 months for people to just start to understand the internals of the software stack at most major companies. And that is working on it everyday, nothing is getting near passing the "Turing Test".

I don’t know what your argument is exactly, but it seems to be something like, “Software engineering is really hard so computers can’t do it.”

A variation of that argument props up most common AI skepticism. I don’t think there’s anything out right now that would convince you, but from what I know, everything you pointed out will be solved within the next few years.

I read the argument as "the hard part of software engineering is understanding the codebase and the world well enough to turn a description of a desired change in how a system should act into a diff that actually changes the system behavior in the intended way (and only in the intended way), not taking clear requirements and turning them into fresh code".

Of course humans aren't exactly great at that part either. But I do think I'd bet against, within the next 4 years, an AI tool being able to take tickets in the form

1. Expected behavior

2. Observed behavior

3. Steps to reproduce

and produce a changelist that legibly fixes the problem, and does not break anything else, at a level better than a typical junior software developer. I think the ability to do that is probably AGI complete.

I think you’re right that code models are a vital research path toward AGI.

The steps you elucidated are all expressible in natural language, and we see models like Codex Edit making headway there. One of the most fascinating parts of this is that once access to the known baselines are provided to high-level engineers, they then go on to do much more than what the models alone can do.

The main hinderance to enterprise was compliance but the move toward Azure, etc, will dissolve those barriers this year.

An AI being able to create arbitrary computer programs and work productively in an arbitrary codebase sounds basically like an AGI or something approximating it. Not sure if that's what you're implying is a few years away. I do think it's inevitable, but the time horizon still doesn't seem clear to me at all

Do you think systems like this will be able to do things like:

- Create a new programming language like Go or Rust?

- Create new infrastructure patterns like containers and kubernetes?

- Build tools like MapReduce/Arrow/Airflow or TF/JAX? Blockchain?

- Understand a new tool or framework it was not trained on?

- Decide when and how to refactor a code base because the mapping to the underlying problem has reached its limits?

- Write or modify compilers for emerging platforms like RiscV or WASM?

- Help to resolve a 1:1000 50x returned by my server?

Do today's ML models come close to the "why" these problems have?

Well I'm certainly looking forward to AI doing the fundamental research needed to model the behavior and work with certain hardware components, or writing clock cycle exact embedded code for complex applications, or debugging a timing problem in an async system... /s

Let's get real, nothing except simple crud apps are getting replaced, if any, anytime soon.

same as GP here. your argument does not make any sense to me.

> Code generation is moving extremely fast

Correction, it was moving fast a few years ago, it isn't moving fast right now. I can understand being a little worried a few years ago, but today? They are clearly stuck, we know the capabilities of current models and they aren't threatening anyone, if it was easy to improve then they would have made huge strides from a couple of years ago but they haven't.

That’s objectively wrong.

Codex, AlphaCode, both surpassed by CodeRL on the challenging APPS benchmark last year. Meta working on InCoder. Microsoft working on UniXCoder…

Future research directions are pretty clear from where we stand. That includes iterative methods, reinforcement learning, text diffusion, etc. No one is stuck.

unixcoder… they had to name it like that, right?

Codex just barely surpassed it on easy questions but did worse on harder ones. AlphaCode is significantly better on harder questions, but significantly worse on easy questions. That isn't extremely fast development, they are mostly moving sideways, trying to improve one part of the metric hurts the others.

https://paperswithcode.com/sota/code-generation-on-apps

Development in these areas was very fast in the 3 years between transformer networks were invented and roughly GPT 3 was done. But in the 3 years since GPT-3 not much has changed, we see a lot of "we applied a large network to a new problem and found X" since then, but that isn't new performance, its just a new result with the same thing we had around back then.

Reply on news.ycombinator.com