Hacker News Reader

Comment by janice1999 | Hacker News Reader

janice1999

JetBrains ships a local, offline AI auto-complete in PyCharm. Sure it's limited, but it shows it can be done. It's a pity other companies aren't trying the same, especially this one that boasts about millisecond editing times.

Edit: JetBrains, not IntelliJ. Auto-complete details - https://blog.jetbrains.com/blog/2024/04/04/full-line-code-co...

View on HN

Zed will hook into ollama as well; so you can run your own locally; just not as part of the same process (yet?).

Hooking into standard APIs like Ollama is definitely preferred (at least to me) because it means it’s more composable and modular. Very Unix-y. Lets you use whatever works for you instead of the vendor.

I tried it with C#. It's not comparable to Copilot.

If I've never used Copilot I might be slightly impressed.

I don’t see programming getting exponentially harder. So eventually local compute will catch up with programming needs and offline AI will be pretty good.

The local compute problem is a cost problem, and most people do not have the hardware necessary and judging by trends this will continue for another ~decade.

Local compute is also getting progressively harder to make faster. I have a CPU from 6 years ago and checked some benchmarks to see if upgrades would be worth it. It's a 90% increase in performance with a 27% improvement in single threaded. Pretty substantial but not exponential. GPUs are advancing more quickly than CPUs right now but I wouldn't be surprised if they hit a wall soon.

For those who don't have the latitude of uploading their code to Microsoft servers, smaller, limited but fully local AI is better than nothing.

You're right, it's not comparable. Jetbrain's code completion actually integrates with the IDE to guarantee the suggested code isn't entirely a hallucination -- for example it checks for unresolved references so that its suggestions don't reference non-existent variables/methods.

I've disabled Copilot as it produces garbage way too often. I found myself "pausing" to wait for the suggestion that I'd ultimately ignore because it was completely invalid. I've left Jetbrain's code completion on though because it's basically just a mildly "smarter" autocomplete that I'll occasionally use / I don't find myself relying on.

I agree that the completions might not be that great, but for context: this is a 100M model, while most models that people compare this to are atleast 100x bigger.

They also focus on single line completions, and ship different models per programming language. All these make it possible to ship a decent completion engine with a very small download size.

Interesting, have you used it/found it to be usable? I use IntelliJ myself, but it's not known for being a lean beast. With the addition of stuff like test containers and a few plugins, I'd be suprised if my machine didn't melt adding a local LLM too.

I mainly use PyCharm and I found the auto-complete to be good. It doesn't always kick in when I expect but some of the suggestions have been surprisingly complex and correct.

Maybe its because there's literally no point in using a local llm for code-completion. You'd be spending 90% of your time correcting it. Its barely worth it to use co-pilot.

AI autocomplete is just a toy, it has been there since the begining of AI code and I find it pretty useless. I prefer making a proper prompt to get better answers. Also most of the time engineers (use AI to) modify existing codebases rather than making new code.

Google's systematic, quantitative evaluation of AI coding assistance tools disagrees with your personal and anecdotal findings (which agrees with many other similar studies):

https://research.google/blog/ai-in-software-engineering-at-g...

I don't see a comparison to Non-GenAI auto complete there.

Eclipse wrote half my code characters 15 years ago.

What programming languages do you mostly work in?

I've been wondering if the benefits of AI-autocomplete are more material to people who work in languages like Python and JavaScript that are harder to provide IDE-based autocomplete for.

If you're a Java or TypeScript developer maybe the impact is reduced because you already have great autocomplete by default.

I think typescript benefits even more than JavaScript because the type signatures add so much context. After defining the types and signatures copilot will often complete the body within a few changes of perfect

Yeah +1, the current top LLMs do pretty well with TypeScript

> AI autocomplete is just a toy

I disagree, when I'm writing code and it just autocompletes the line for me with the correct type with the correct type params set for the generics it saves me the mental effort having to scroll around the file to find the exact type signature I needed. For these small edits it's always right if the information is in the file.

Why do you need an LLM for that? That basic type inference.

i dont know what statically typed language you are using that never requires specifying types. As far as i know only ocaml works this way.

Different languages support different levels of type inference. When im writing rust and typescript im often specifying types this way for structs and function signatures.

With llms what often happens is i write a parameter or variable name and it infers from the name what the type is because i have used the same name somewhere else in my codebase. This would not work without llm autocomplete

What it shows is that it can be done — in a limited way. Other people might not like those limits and chose to go a different way. I am not sure what's worth lamenting here.

> I am not sure what's worth lamenting here.

The normalisation of surrendering all our code to remove AI cloud gods maybe? The other being a super responsive IDE now having major features have network requests delaying them, although HW requirements likely make that faster for most people.

That sounds a little too spooky for my taste, but you do you. What anything beyond that means, in effect, is that you (not necessarily you) want to chose my values for me (not necessarily me).

I don't see why I should care to have you do that.

Zed is trying to position itself as a next gen hackers editor. Text editing with lessons learned from the multitude of experiments in the space. A flagship feature that is online only and requires me to trust a third party is a deal breaker to many. For instance, my employer would not be pleased if I suddenly started sharing code with another.

Take the AI out of the conversation: if you told your employer you shared the codebase, that’s an insta-fire kind of move.

Yeah but so is I sent a message to s random person about our business vs I sent a message to one of our suppliers

Is Zed a supplier? Sounds like a random developer can sign up at will without any corporate NDAs or other established legal agreements. Will my employer hash out the NDAs for every developer tool that wants to incorporate cloud AI for the lulz?

What's the impact on laptop battery life roughly, when you have a local LLM constantly running in the background of your text editor?

IntelliJ is an IDE, though, not a text editor. If you want a text editor with AI, you may need to wait for Microsoft to bring ChatGPT to notepad.

It seems to be limited to a single CPU core at a time, so depending on your CPU boosting settings, some but not too much.

It's quite snappy despite not using a lot of resources. I tried replicating the effect using a CUDA-based alternative but I could get neither the snappiness nor the quality out of that.

sure, but if it was good more people was use it

running a much worse model at a higher latency (since local GPU power is limited) is a worse experience for Zed.

Reply on news.ycombinator.com