Hacker News Reader

Comment by muglug | Hacker News Reader

GitHub touts a 55% improvement in coding speed based on a study conducted in-house that tested the participants’ ability to paste a pre-written prompt and then check the output: https://github.blog/2022-09-07-research-quantifying-github-c...

The effectiveness of a Copilot-like tool trialed at FB showed that 8% of code contributed by participants was sourced from suggestions, but the latter study made no promise about coding velocity: https://arxiv.org/abs/2305.12050. In my own experience the time taken to review machine-generated suggestions often eats into developer time.

This is not a critique of LLMs in general — I’ve found ChatGPT really great for kicking off greenfield projects in well-known languages and frameworks.

View on HN

My personal feeling is that utilising LLM assistance often isn't faster, but it can take less "stamina", tiring me less.

Especially when renaming variables that are “immune” to normal refactoring. Copilot handles that pretty well and I don’t have to spend all that focus on such a menial task.

Yes, I often find that copilot is pretty good at picking up on the pattern of refactorings that I am doing.

Especially those that are a bit tedious and almost mechanical, but not quite mechanical enough to do with a simple search-and-replace.

Vim and text motions should do the trick.

Sure, but doing the refactor the first time, without thinking about how to record a vim macro, then just hitting tab to have copilot do the same change over and over is a lower friction experience

Totally depends on what kind of refactoring you want to do; and how well vim's commands map to your language's syntax, too. (Or whether your vim has special support for your language's syntax.)

Sidebar, but what does "“immune” to normal refactoring" mean?

I would guess they mean stuff which isn’t just “rename symbol” or whatever, like changing code from one pattern to another slightly different one. For example, I’ve used LLMs to “change this if statement to a switch”, which I don’t think VS Code can do as an automatic refactor using the native tools.

That’s not variables, though. But maybe it’s something similar, like some variables being used in strings?

> which I don’t think VS Code can do as an automatic refactor using the native tools.

Yet another point for using an IDE over a texteditor with some bolted on IDE features, because the Jetbrains tools even suggest such refactorings ;)

That's a great point. I'm definitely procrastinating less.

Yeah although the frustration of restarting after failed & failed attempts, or introducing bugs in parts of code you didn't want updated is also tiring.

I've had the opposite experience - copilot will pop up a suggestion that's wildly wrong and I'll expend energy processing and discarding it.

I find it far more tiring because I don't have "micro-breaks" where I'm slowly typing code. I just have to be in a serious "check the logic" mode for a longer period of time.

It also makes it a lot easier for juniors to chuck random code at seniors for review.

I have a "top class" LLM based autocomplete provided by work.

At first it was a massive pain because I didn't realise it wasn't a "proper" autocomplete(intellisense is probably the king in that regard), and get hit with a large number of hallucinated functions.

This was really hard for me, as I'm slightly dyslexic, which means spotting plausible but bullshit completions is very hard. (I suspect its hard for everyone else too). Worse still, at the time the linter/type inspector was/is very slow so only ran on save/execution.

However its both improved in the last year significantly, and I have got used to it. For me there are a few techniques that help me:

1) It has a recency bias. Which is great for when you're jumping about in code making changes

2) It rewards proper variable names

3) Your comments should say what, and why your doing what you are doing.

2 & 3 should be obvious and you should be doing it anyway. But it really re-enforces that.

However I would really like some UI changes so that I can make _better_ use of the LLM plugin.

1) a completely different colour to indicate that its an LLM suggestion (bonus points for giving a confidence as well)

2) a different keystroke to accept the suggestion. (bonus for partial selection)

Coding speed really is a horrible metric. I can code really quickly, but it doesn't mean I'm doing anything productive or correct.

I'd rather slower coding speed, properly written as it provides higher overall velocity. And velocity should take into account the refactoring that happens months or even years later. Crappy code can look really fancy, and even be bug free, but if it's overengineered and hard to change, it can create long change times or even a full stop in development later in the products lifecycle.

And that's on top of developers losing the understanding of how something actually works. If AI helps create the code that would have been written without AI, then great, but I don't observe that happening, and the code has never been a better idea than then dev could have done without it.

This last sentence is the most important. You can describe your dB schema in words and get laravel migration, models, controllers and if you wanted policies, form requests etc all near perfect. You can get a V1 in 10-20 minutes then go about handling all the actual logic

Yeah the copilot doesn’t need to be integrated into every keystroke, just able to analyze the context and kickstart the code. Getting up to speed with new libraries is so much easier with AI instead of the bad old days of trial-and-error-and-marked-as-duplicate

Can Copilot analyze my whole codebase these days or just the file I've opened? Honest question as I've stopped to use it a long time ago.

It seems to be hit and miss, at least with the current PyCharm integration. In some cases it can infer information using other files, in some cases it can't (even if they are open in other tabs).

As someone who switches between PyCharm and VSCode, I find that Copilot seems to work better in VSCode for some reason. Nothing major, but the suggestions I get just seem slightly more relevant to my code base, and are more often what I wanted.

Although I could be hallucinating the whole thing.

Whole-codebase understanding is beyond current coding assistants’ capabilities. To do that you need to add in traditional static analysis. But I’m sure people are working on it!

With VSCode at least you can say @workspace to Copilot and it'll take it into account.

No idea if it feeds all of the code to the LLM or just parts, but it's pretty good at interpreting what the code does

It guaranteed can't have all of your code in the LLM context, or even all of your file (in case of longer, through not even quite long files).

Through it could do stuff like go through your repository(ies) and generate embedding for sections of it and then have a vector database + retrieval argumented generation (RAG) system.

Yea, that's the way they're going based on the low-memory models they're building.

Basically all the LLM needs to do is translate human writing to some format that a "normal" service can use. That can then leverage the existing spotlight system that's pretty decent at searching stuff on the phone anyway.

Then it'll report it back to the LLM which translates whatever format back to something humans can process.

Basically "less shit Siri"

A lead from one of the GH Copilot-adjacent companies was interviewed recently, and that’s precisely what they are doing. They generate embedding of “local” code based on AST (up the stack and sideways, if you know what I mean), and take into account runtime and library versions when doing inference. Sounded like a very interesting challenge.

I especially like when I'm looking for a particular method on a class I've never used before and it just makes one up for me as though it exists.

It’s adding features that should have existed for you! It can also make up the documentation for the method while you are at it.

So rails scaffold from 20 years ago but with a chance of hallucinations?

Shhh we’re raising trillions for this technology remember?

It's very easy to find multiple solutions to one problem. The power of LLM is that it's one solution to many problems.

But is it actually a solution to any of them?

The "chance of hallucinations" is the tricky bit - if I have to manually check everything it does in case it's hallucinating, then it's not actually a solution. It's not saving me time (as TFA says).

while you’re in doubt, i’ll be speeding out

Speed on, I'll enjoy the slo-mo crash footage later :)

What about the problem of needless pollution?

I thought we had moved on from cryptocurrencies

The last entity I would trust for trustworthy data on how much Copilot helps programmers is GitHub. They obviously skew the scenarios and metrics to find the best possible number to report.

I think the truth as revealed in your comment and many others is that it all depends on the context. For repetitive code or boilerplate, or starting new projects in well-known frameworks and languages, it probably does increase velocity. The other context factor is the programmer themselves. Their familiarity with the problem domain, the language, and the framework all matter, as does the personality and coding style of the individual.

It also depends on what tools a developer is already using as a productivity booster.

For example, I use vanilla web components which have some boilerplate for new components.

I already have a simple vscode snippet that does the job well, with no hallucinations [1]. I've experimented with llms doing the same thing with not great results.

It took me longer to explain what I wanted than it did for me to just write that snippet. Doing it repeatedly and waiting for the results definitely didn't increase my speed, though I was impressed that it was eventually able to figure it out (vanilla web components with lit-html renderer isn't a super common technique). Also, I prefer the pythonic approach of snake-case local variable names. Getting the llm to do that in a js project where it's not super common was another whole iteration.

We've had code generation tools around for decades to deal with repetitive boilerplate tasks. Maybe they aren't quite as capable as when the llm "gets it right" - but I wonder with these productivity claims how they are measured.

Are they starting from zero and a new developer? Or comparing against an experienced developer with proficiency with lots of workflow enhancers like templates, snippets, vim macros, etc...

[1] https://gist.github.com/claytongulick/e4251c1b27b22c25fb68f3...

So I am a scientist and I've generally resisted using llms for my work. My general critique reading others' experience is this sounds like what is needed is better abstractions if all it helps people with is doing boilerplate that needs to be checked. Also no need for "better abstractions" to be some ethereal thing, this just means better standardised libraries and frameworks. May be if the word "abstractions" sounds too bespoke may be what is needed is better tools and tooling.

The standardised bit seems more difficult than it should be as it's essentially a social problem with development in general, but that also is a problen AI does not address, it's merely a bandaid over the problem.

It takes literally 10 minutes to try it out. I'd suggest trying it instead of recycling other people's opinions.

Reply on news.ycombinator.com