Hacker News Reader

Comment by ccvannorman | Hacker News Reader

Just because this article isn't well formed or sourced doesnt make its claim incorrect.

I program daily and I use AI for it daily.

For short and simple programs AI can do it 100x faster, but it is fundamentally limited by its context size. As a program grows in complexity AI is not currently able to create or modify the codebase with success (around 2000 lines is where I found it has a barrier). I suspect it's due to exponential complexity associated with input size.

Show me an AI that can do this for a 10,000 lines complex program and I'll eat my own shorts

View on HN

Doesn't even take that much. Today I did some basic Terraform with the help of GenAI - it can certainly print out the fundamentals (VPC, subnets) faster than I can type them myself, but the wheels came off quickly. It hallucinated 2 or 3 things (some non-existent provider features, invalid TF syntax, etc).

When you take into account prompt writing time and the effort of fixing its mistakes, I would have been better off doing the whole thing by hand. To make matters worse, I find that my mental model of what was created is nowhere near as strong as it would have been if I did things myself - meaning that when I go back to the code tomorrow, I might be better off just starting from scratch.

Here's a thought experiment. Think back on how that statement would have sounded like to past-you, 3 years ago. You would probably have dismissed it as bullshit, right? We've gone a long way since then. Both in terms of better, faster and cheaper models, but also how they're being intertwined with developer tooling.

Now imagine 3 years from now.

As you inadvertently pointed out, AI improvements are not linear. They depend on new discoveries more than they do iteration. We could be in either out of jobs or lamenting the stagnation of AI (again).

After an innovation phase there is an implementation phase. Depending on the usefulness of the innovation, the integration with existing systems takes time. It is calculated in years, tens of years. Think back on the 80-90s, where it took years to integrate PCs into offices and workspaces.

From your comment, is sounds like you think that the implementation phase of LLMs is already over? And if so, how do you come to this conclusion?

It's not as if we have no idea how to make use of AI in programming. We've been working on AI in one form or another since the 70's, and integrated them with our programming workflow almost as long (more recently in the form of autocomplete using natural language processing and machine learning models). It's already completely integrated into our IDEs, often with options to collate the output of multiple LLMs.

What further implementations of integrating AI and programming workflows have LLMs shown to be missing?

You could have said the same for crypto/blockchain 3-4 years ago (or whenever it was at peak hype).

Eventually we realized what is and isn't possible or practical to use blockchain for. It didn't really live up to all the original hype years ago, but it's still a good technology to have around.

It's possible LLMs could follow a similar pattern, but who knows.

What good thing has blockchain ever done that isn't facilitating crime or tax evasion?

It created a speculative asset that some people are passionate about.

However, if you saw the homepage of HN during blockchain peak hype, being a speculative asset / digital currency was seen almost as a side effect of the underlying technology, but it turns out that’s pretty much all it turned out to be useful for.

You can imagine all sorts of things, and then something else might happen. You can’t rely on “proof by imagination” or “proof by lack of imagination.”

We shouldn’t be highly confident in any claims about where AI will be in three years, because it depends on how successful the research is. Figuring out how to apply the technology to create successful products takes time, too.

Ten years ago when Siri/Google/Alexa were launching, I really wouldn't have expected that 2024 voice assistants would be mere egg timers, and frustrating ones at that - requiring considered phrasing and regular repeating/cancelling/yelling to trick it into doing what you want.

A 10x near future isn't inconceivable, but neither is one where we look back and laugh at how hyped we got at that early-20s version of language models.

It is a great point.

It also might be that the language everyone uses 20 years from now that gives a 50X from today is just being worked on right now or won't come along for another 5 years.

The way people who would have thought that humans could never fly were not completely wrong before the airplane. After the airplane though, we are really talking about two different versions of a "human that can fly".

Or you are extrapolating from the exponential growth phase in a sigmoid curve. Hard to say.

GPT-4 has been out for 1.5 years and I haven't seen much improvement in code quality across LLM's. It's still one of the best.

Same thing that can be said about autonomous cars in 2014?

Not everything will grow exponentially forever

In my very uninformed opinion, all we need is more clever indexing, prompting, and agents that can iteratively load parts of the codebase into their context and make modifications.

Real engineers aren’t expected to hold 10,000 lines of exact code in their head, they know the overall structure and general patterns used throughout the codebase, then look up the parts they need to make a modification.

I suspect it's due to exponential complexity associated with input size.

I am curious how you get to exponential complexity. The time complexity of a normal transformer is quadratic.

Or do you mean that the complexity of dealing with a codebase grows exponentially with the length?

Generally speaking, complexity grows exponentially with input size.

A program of length 100 lines has the potential to be 1000x as complex as a program with length 10 lines.

Consider the amount of information that could be stored in 4 bits vs 8 bits; 2^4 vs 2^8.

As the potential to be complex grows, so does current AI's ability to effectively write and modify code at that scale.

I would recommend giving a try to o1-preview in coding tasks like this one.

It is one level above Claude 3.5 Sonnet, which currently is the most popular tool among my peers.

Reply on news.ycombinator.com