Hacker News Reader

Comment by muglug | Hacker News Reader

Yup — this doesn't match my experience using Rust with Claude. I've spent 2.5 years writing Rust professionally, and I'm pretty good at it. Claude will hallucinate things about Rust code because it’s a statistical model, not a static analysis tool. When it’s able to create code that compiles, the code is invariably inefficient and ugly.

But if you want it to generate chunks of usable and eloquent Python from scratch, it’s pretty decent.

And, FWIW, I’m not fluent in Python.

View on HN

> Claude will hallucinate things about Rust code because it’s a statistical model, not a static analysis tool.

I think that's the point of the article.

In a dynamic language or a compiled language, its going to be hallucinating either way. If you vibe coding the errors are caught earlier so you can vibe code them away before it blows up at run time.

Static analysis tools like rustc and clippy are powerful, but there are large classes of errors that escape those analyses — e.g. things like off-by-one errors.

> If you vibe coding the errors are caught earlier so you can vibe code them away before it blows up at run time

You can say that again.

I was looking into the many comments for this particular comment and you did hit the nail on the head.

The irony is that it took the entire GenAI -> LLM -> vibe coding cycle to settle the argument that typed language is better for human coding and software engineering.

Sure, but in my experience the advantage is less than one would imagine. LLMs are really good at pattern matching and as long as they have the API and the relevant source code in their context they wont make many/any of the errors that humans are prone to.

Hah... yeah, no, its Python isn't great. I'd definitely workable and better than what I see from 9/10 junior engineers, but it tends to be pretty verbose and over-engineered.

My repos all have pre-commit hooks which run the linters/formatters/type-checkers. Both Claude and Gemini will sometimes write code that won't get past mypy and they'll then struggle to get it typed correct before eventually by passing the pre-commit check with `git commit -n`.

I've had to add some fairly specific instructions to CLAUDE.md/GEMINI.md to get them to cut this out.

Claude is better about following the rules. Gemini just flat out ignores instructions. I've also found Gemini is more likely to get stuck in a loop and give up.

That said, I'm saying this after about 100 hours of experience with these LLMs. I'm sure they'll get better with their output and I'll get better with my input.

To be fair, depending on what libraries you’re using, Python typing isn’t exactly easy even for a human, I spend more time battling with type checkers and stubs than I would like.

I can confirm input matters a lot. I'm a couple of hundred hours ahead of you and my prompting has come along a lot. I recommend test cycles, prompts to reflect on product-implementation fit (eg, is this what you've been asked to do?) and lots of interactivity. Despite what I've written elsewhere in these comments, the best work is a good oneshot followed by small iterations and attentive steering.

With access to good MCP tools, I've had really good experience using claude code to write rust: https://news.ycombinator.com/item?id=44702820

What MCP tools are you using?

Honestly, it's mostly just some random LSP adapter I forked and fixed a few bugs on, and it's not even that comprehensive but it goes a long way and seems most essential. Then I have some notes in the long term context about how to use a combination of gh CLI and cargo docs to read documentation and dependency source code/examples.

A few things beyond your question, for anyone curious:

I've also poked around with a custom MCP server that attempts to teach the LLM how to use ast-grep, but that didn't really work as hoped. It helps sometimes but my next shot on that project will be to rely on GritQL. Smaller LLMs stumble over the YAML indentation. GritQL is more like a template language for AST aware code transformations.

Lastly, there are probably a lot of little things in my long term context that help get into a successful flow. I wouldn't be surprised if a key difference between getting good results and getting bad results with these agentic LLM tools is how people are reacting to failures. If a failure makes you immediately throw up your hands and give up, you're not doing it right. If instead you press the little '#' (in claude code) and enter some instructions to the long term context memory, you'll get results. It's about persistence and really learning to understand these things as tools.

Interesting, so an LSP MCP like this?: https://github.com/isaacphi/mcp-language-server

Also interesting note on the docs, though, Claude does try to use cargo doc by itself sometimes.

I was actually wondering why GritQL did not have an MCP, this seems like a natural fit. Would be interested to know if this works for you.

I'm always a bit hesitant to add things to the long term context as it feels very finicky to not have it be ignored and having more seems to make it more likely to be ignored. Instead I usually just repeat myself.

Thank you for the answer, seems there is still lots of things to try.

> When it’s able to create code that compiles, the code is invariably inefficient and ugly.

Why not have static analysis tools on the other side of those generations that constrain how the LLM can write the code?

I'd be interested to know the answer to this as well. Considering the wealth of AI IDE integrations, it's very eyebrow-raising that there are zero instances of this. Seems like somewhat low hanging fruit to rule out tokens that are clearly syntactically or semantically invalid.

This can be done: I gave mine a justfile and early in the project very attentively steered it towards building out quality checks. CLAUDE.md also contains instructions to run those after each iteration.

What I'd like to see is the CLI's interaction with VSCode etc extending to understand things which the IDE has given us for free for years.

I’d like to constrain the output of the LLM by accessing the probabilities for the next token, pick the next token that has the highest probability and also is valid in the type system, and use that. Originally OpenAI did give you the probabilities for the next token, but apparently that made it easy to steal the weights, so they turned that feature off.

It's been tried already and doesn't work. Very often a model needs to emit tokens that aren't valid yet but will become so later.

> Why not have static analysis tools on the other side of those generations that constrain how the LLM can write the code?

We do have it, we call those programmers, without such tools you don't get much useful output at all. But other than that static analysis tools aren't powerful enough to detect the kind of problems and issues these language models creates.

LLMs are famously bad at producing rust code. I'm not sure how much of it is the lesser amount of Rust code in the training data, or just the fact that Rust has a very large number of pitfalls, and a large standard library with many edge cases and things you'd imaging should exist but don't for a variety of reasons. Rust also has a much wider variety in the way things could be structured, compared to something like go where there is often only one way of doing a particular thing.

Honestly, I don't think these are problems that Rust has. What I see LLMs struggle with in Rust is more to do with understanding the language semantics at a fundamental level - exactly the things that the compiler statically verifies. For example, they will see things they think are "use-after-free" or "use-after-move", neither of which is a thing in (safe) Rust, because they don't understand that the language does not have these problems.

Largely I think LLMs struggle with Rust because it is one of very few languages that actually does something new. The semantics are just way more different than the difference between, say, Go and TypeScript. I imagine they would struggle just as much with Haskell, Ocaml, Prolog, and other interesting languages.

Obviously you can write a use-after-free in Rust. The fact that it won't compile doesn't really matter when you're feeding the text to a non-compiler program like an LLM. I trust you don't mean to get carried away and suggest that they're somehow grammatically impossible.

I feel like I have had just as much luck with LLMs writing Rust as I have had with Java, Kotlin, and Swift. Which is better than C++ and worse than Python. I think that mostly comes down to the relative abundance of training data for these types of codebases.

But that is all independent of how the LLMs are used, especially in an agentic coding environment. Strong/static typed languages with good compiler messages have a very fast feedback loop via parsing and typechecking, and agentic coding systems that are properly guided (with rulesets like Claude.md files) can iterate much quicker because of it.

I find that even with relatively obscure languages (like OCaml and Scala), the time and effort it takes to get good outcomes is dramatically reduced, albeit with a higher cost due to the fact that they don't usually get it right on the first try.

I have had very good results using Claude to write Rust. My prompting is often something like

'I have a database table Foo, here is the DDL: <sql>, create CRUD end points at /v0/foo; and use the same coding conventions used for Bar.'

I find it copies existing code style pretty well.

> When it’s able to create code that compiles, the code is invariably inefficient and ugly.

At the end of the day this is a trivial problem. When Claude Code finishes a commit, just spin up another Claude Code instance and say "run a git diff, find and fix inefficient and ugly code, and make sure it still compiles."

Reply on news.ycombinator.com