Hacker News Reader

Comment by Closi | Hacker News Reader

> I think it's as close to the perfect benchmark as you can get.

Depends what you are benchmarking for... If you are benchmarking the ability of the solution to solve LEETCODE challenges, that is different to the ability of GPT4 to assist everyday programmers knock out business logic or diagnose bugs.

My experience of GPT4 is that it's significantly better at the latter than GPT3.5.

Additionally, the real test is for me is "Can an average programmer using GPT4 as a tool solve Advent of Code faster than an equally-skilled programmer without an LLM?".

View on HN

>diagnose bugs

Literally says in the article that GPT's main drawback is that it can't debug in the same way a human can.

It can if you use the python interpreter.

Fixing bugs in code it writes itself is very different to diagnosing (i.e. identifying, not fixing) bugs in my own code.

When stuck I often paste code into ChatGPT and ask why it doesn't work, and it will often help me quickly identify the error and propose a fix.

I have done the same thing. “I am seeing the following behavior, but I expected <X>. What am I missing?” That line has often quickly solved some bugs that would have otherwise taken some time to debug. Very handy!

Can you share some examples of this? I haven’t had much luck with ChatGPT correctly identifying issues because (in my case, at least) they stem from other parts of a large codebase, and (last time I checked) I couldn’t paste more than a few kilobytes of code into ChatGPT.

One example are bugs caused by precondition violations, which ChatGPT can’t diagnose without also being given the code to all of the incoming call-sites, which means you end-up solving the problem yourself before you’ve even explained the issue to ChatGPT - so (to me, at least) my use of ChatGPT is more akin to rubber-duck-debugging[1] than anything else.

[1]: https://en.wikipedia.org/wiki/Rubber_duck_debugging

> Fixing bugs in code it writes itself is very different to diagnosing (i.e. identifying, not fixing) bugs in my own code.

yeah, we run the random number generator again and hopefully this time less buggy code pops out

Consider for a moment the possibility that TFA is wrong

> Can an average programmer using GPT4 as a tool solve Advent of Code faster than an equally-skilled programmer without an LLM?

I mean, yes, obviously? Any tool, even if it had near-zero marginal utility must necessarily improve performance, as you can simply elect not to use it.

The disagreement is not in the direction, it's in the magnitude. Therefore the real test is: "Can an average programmer solve AoC faster with GPT4 and without syntax coloring or with syntax coloring and without GPT4?", or "GPT4 in C versus no-GPT4 in Python", or "GPT4 on a crappy laptop vs no-GPT4 on a high-end workstation" and so on.

Why are your tests changing multiple variables?

A simpler test would be keeping the development environment the same and then adding GPT4 to see if there is a statistically significant and meaningful speed increase (of a decent magnitude).

I'm not looking for a 1% speed improvement, I'm looking for a >50% speed improvement. Maybe I should have stated 'significant' speed improvement in the initial post.

It's not meant to be a scientifically rigorous test, but the idea is this.

It seems like you're accepting that (at least in its current state, barring unknowable future discoveries) this is a technology that complements developers, making them more productive.

If this is the case, then the question is how relevant this improvement is in quantitative terms. The cumulative improvements in developer productivity since the days of punching cards have easily been several hundred percent.

A thought experiment to figure out how relatively impactful this is would be to compare it to other technologies and see which gives the greatest boost.

My prior belief: somewhere around Intellisense level of useful, but not significantly more than that.

Yes, I think the current state of technology (just starting) is a technology that complements developers.

Although my personal belief is this is more likely going to be an improvement similar to Assembly -> C (even if the LLM component doesn't improve, but assuming that the tooling does improve).

Personally I think there is going to be a new 'higher, higher level' of programming paradigms that are about to be invented that are supported by the ability of LLM's to write code - still augmenting humans, but making them many times more productive.

I think the opposite - LLMs will cause programming language development to become stagnant because the language that compliments an LLM the best is the language that has the most code out there in the wild.

So the "LLM language" already exists - its Javascript, Java, C#, etc.

I'm not talking about a language that an LLM outputs.

I'm talking about a higher-level language that outputs Javascript, Java, C# etc (but presumably some sort of lower level byte-code in the future).

A language where intent and clarity is more important than syntax or implementation details.

Where

menu_options = [process['name'] for process in processes if process['active']]

is more clearly expressed as

menu_options = list of the active process names

(or something similar)

> Any tool, even if it had near-zero marginal utility must necessarily improve performance, as you can simply elect not to use it.

That might be true for any tool with consistent, predictable output that you thoroughly understand. Someone could easily lose a few hours dickering around with chatgpt only to realize they’re better off just starting it from scratch.

> Any tool, even if it had near-zero marginal utility must necessarily improve performance, as you can simply elect not to use it.

No opinion on the specific tradeoff here, but in general it's not obvious to me that your statement is true. Using tools involves opportunity cost, so sometimes "not electing to us it" can on average be a net win, no? There is a cognitive and time load to using it at all, right?

This reminds me of a study where they compared the time taken to solve a problem comparable to a two-star AoC problem in difficulty across different dev environments, such as languages and IDEs.

There are a lot of anecdotes about how "it's the programmer, not the language" or "any language can be productive", etc...

The actual results were that many popular languages are more than an order of magnitude slower to achieve time-to-correct-solution than others. From memory, the fastest was F# with about 20 minutes, then Python and C# with about 40 minutes for both, and then C/C++/Java were hours to even days!

Do you have a link to that study? I'd be very interested in the full results!

Reply on news.ycombinator.com