Hacker News Reader

Comment by prmph | Hacker News Reader

> LLMs are remarkably good at writing code.

Just this past weekend, I've designed and written code (in Typescript) that I don't think LLMs can even come close to writing in years. I have a subscription to a frontier LLM, but lately I find myself using like 25% of the time.

At a certain level the software architecture problems I'm solving, drawing upon decades of understanding about maintainable, performant, and verifiable design of data structures and types and algorithms, are things LLMs cannot even begin to grasp.

At that point, I find that attempting to use an LLM to even draft an initial solution is a waste of time. At best I can use it for initial brainstorming.

The people saying LLMs can code are hard for me to understand. They are good for simple bash scripts and complex refactoring and drafting basic code idioms and that's about it.

And even for these tasks the amount of hand-holding I need to do is substantial. At least Gemini Pro/CLI seems good at one-shot performance, before its context gets poisoned

View on HN

I found that mastering LLM is no less complex than getting to learning a new language, probably between python and C++ in terms of mastery.

The learning curve is very different - with other languages, the learning curve is often upfront, with LLM, it seems linear/even rear loaded, maybe because I've not gotten to the other side.

I've been able to make LLM do more and more, some of it is undoubtly due to the improvement in model, but most of it is probably paradigm and changes in my approach. At the beginning, I run into all of the same complaints that I have eventually found workarounds to many.

> The people saying LLMs can code are hard for me to understand. They are good for simple bash scripts and complex refactoring and drafting basic code idioms and that's about it

that's like, 90% of the code people are writing

But not 90% of the work people do. It’s solved a task, not a problem.

It's what takes time though. When you need to make a wrapper for some API for example LLMs are incredible. You give it a template, the payload format and the possible methods and it just spits out a 500-1000 line class in 15 seconds. Do it for 20 classes, that's work for a week 'done' in 30 mins. Realistically 2 days since you still have to fix and test a lot but still..

Or write a lisp macro in one hour and be done. Or install an opengenerator and be done in 10 minutes, 9 of which is configuring the generator.

If you can get the specific documentation for it. Sadly many companies don't want you using the API so they just give you a generic payload and the methods and leave you to it. LLMs are good in the sense that they can tell what type StartDate, EndDate is (str MSDate), maybe it also somehow catches on that ActualDuration is an int.. It also manages to guess correctly a lot of the fields in that payload that are not necessary for the particular call/get overridden anyway.

Can a Lisp macro automatically search for, and find, the API documentation and apply it to the output?

I've implemented connections to (public) APIs of different services multiple times using LLMs without even looking up the APIs myself.

I just say "Enrich the data about this game from Steam's API" and that's about it.

I find LLMs most helpful when I already have half of the answer written and need them to fill in the blanks.

"Take X and Y I've written before, some documentation for Z, an example W from that repo, now smash them together and build the thing I need"

This is so true. I've had the same experience.

I think C# is really going to shine in the LLM coding era. You can write Roslyn Analyzers to fail the build on arbitrary conditions after inspecting the AST. LLMs are great at helping you write these too. If you get a solid architecture well defined you can then use these as guardrails to constrain development to only happen in the manner you intend. You can then get LLMs to implement features and guarantee the code comes out in the shape you expect it to.

This works well for humans too, but custom analysers are abstract and not many devs know how to write them, so they are mostly provided by library authors. However, being able to generate them via LLMs makes them so much more accessible, and IMHO is a game changer for enforcing an architecture.

I've been exploring this direction a lot lately, and it feels very promising.

Can you expand a little? What you’re suggesting sounds a bit like program verification, or at least program analysis. But what properties are you checking?

I have written many program analyses (though never any for C#; I’ll have to check it out), and my experience is that they are quite challenging to write. Many are research-level CS, so well outside the skill set of your average vibe coder. I’m wondering if you have some insight about LLM generated code that has not occurred to me…

I'm looking at AST based tools in Python to lint, enforce modularity, ban certain patterns. LLMs allow me to write scripts to find recursive function calls, calling super().method() in overrides etc.

https://pypi.org/project/import-linter/ https://github.com/hchasestevens/astpath

I do quite a bit of coding in C#, and have a lot of experience, and personally I haven't found LLMs to be that great a help at writing C#.

First, LLMs are great at learning new tech stacks, but good ol' ASP.NET has been pretty much stable since forever. Second, I think Rider/Resharper is the greatest piece of autocomplete tech ever made, seriously nothing ever comes, close, which means I'd rather do a refactor using them than do something similar by prompting the AI and hoping for the best. Also probably my experience makes me far less accepting of LLMisms, but that might just be on me.

Lastly, AI seems to be focused around its own set of tooling, like Cursor, which is fine for TS but is far worse than Rider for things like C#. I know I could kludge things together, but still.

As for Roslyn...

I have some experience writing codegen/analyzers at my company and it feels like typical a Microsoft tech product, like WPF or Powershell.

Brilliant idea (that's a market first as well) combined with really solid technical fundamentals, but plain confusing and overcomplicated UX, that makes it a chore to use. Seriously the amount of scaffolding you need to make even for a simple analyzer is just nuts

My point is I can my an analyser in like 20 minutes now and it's not a chore at all. I've made like 15 for my current codebase, and when the LLM goes off the rails and generates code in a pattern I don't like, I don't tell it how to write the code much anymore, I tell it to write an analyser that prevents it from writing the code that way and then it goes about fixing it up because now the build fails.

I do a lot of coding in C# with Rider, and refactoring has been a career speciality of mine. I personally find LLMs to have a tonne of value in this space.

> Lastly, AI seems to be focused around its own set of tooling, like Cursor

Nah, the best coding LLMs are console applications like Claude Code, Codex CLI and the like.

Editor integration mostly brings more tools, like tapping into different validators on VSCode and examining the "problems" view.

Also Rider's autocomplete is at least partially AI powered unless you specifically disable it IIRC.

Why I never bothered writing one is the scaffolding, and the dumb idea to write code with WriteLines instead of a nice experience like T4 templates.

Completely agree, and I've started writing more Roslyn analyzers to provide quick feedback to the LLM (assuming you're using it in something like VS Code that exposes the `problems` tool to the model).

I also want C# semantics even more closely integrated with the LLM. I'm imagining a stronger version of Structured Model Outputs that knows all the valid tokens that could be generated following a "." (including instance methods, extension properties, etc.) and prevents invalid code from even being generated in the first place, rather than needing a roundtrip through a Roslyn analyzer or the compiler to feed more text back to the model. (Perhaps there's some leeway to allow calls to not-yet-written methods to be generated.) Or maybe this idea is just a crutch I'm inventing for current frontier models and future models will be smart enough that they don't need it?

Library authors don’t really provide custom analyzers. heck, the best we can hope for are some regex based linting rules, anything that involves local data flow analysis is very rare, and anything inter procedural is non-existent. Program analysis is a dark hole, you are better off just making stronger type systems, but then type inference starts to bite you if you want to support it (and you will given how annoying type annotations are to write, unless you go with something simple like a purely structural type system so you can use Hindley Milner).

Analyzers are a first class citizen in C#. You can get access to the AST during compile time and use it to output diagnostics with error or warning level, so it's more robust than just regex.

I've not seen teams personally write them because they're abstract and most devs shy away from it, but working in the C# ecosystem the one place I do see them pop up occasionally is from library authors.

For example xunit has some.

https://github.com/xunit/xunit.analyzers

So far I have found the value in them to be they help you constrain the possible valid moves that can be made in a codebase. This is valuable with teams of human engineers, but even more so with LLMs. It just so happens LLMs are really good at helping you write them too given you know what you want to enforce.

I doubt the tooling is as good in other languages as it is in C# in this respect, but at least for devs working in the C# ecosystem LLMs have unlocked access to writing custom analysers on a whole new level, and with that it's now significantly easier to define and enforce rules regarding what constitutes a valid program, such that the set of valid programs matches your intended architecture.

As someone whose C# is one of the main work ecosystems, I highly doubt it.

What I am seeing it that LLMs will push current programming languages down the stack, like now you're enjoying C# => MSIL => Machine code.

On my line of work I already can imagine the other side of the tunnel, more low-code/no-code tooling, orchestration agents, and much (much) less manually writing C#, Java and TypeScript.

I don't disagree with your take. My take on your take is that via what I'm suggesting I can envision that the low-code/no-code tooling can be expanded to produce a wider variety of more flexible programs with robust, consistent C# code underpinning them.

Linters are great at catching specific pattern violations, but they’re useless against bad decomposition or a poorly chosen abstraction. An LLM can generate code that passes all 100 linters and still ends up being a logical mess - with business logic in the wrong layer and completely unmaintainable.

With Roslyn Analyzers and things like ArchUnit I've found its possible to actually write linters that enforce a predetermined architecture with established patterns such as enforcing usage of particular base or framework level classes in specific layers/locations.

I agree with the assessment LLMs aren't great at novel architectural work. I'm merely reporting my experience that using LLMs to write analysers that enforce established patterns takes the output from random to well ordered and provides a nice productivity boost in a constrained, but valuable, set of circumstances. It's not a complete solve, but it's a big improvement over just prompting them and hoping for the best.

This approach is perfect for mature projects with a well-established architecture. But it could be counterproductive in the early R&D stages when the architecture is still fluid and the team is constantly experimenting. The rigid constraints of the analyzers might stifle creativity. So it's a powerful tool, but for the right stage of a project's lifecycle

Exactly this. LLMs may do a passable job of architecture if there are many examples of high quality architecture similar to what you want to do in their training set, but to introduce some novel stuff and they are clueless

Can you maybe give an example you’ve encountered of an algorithm or a data structure that LLMs cannot handle well?

In my experience implementing algorithms from a good comprehensive description and keeping track of data models is where they shine the most.

Example (expanding on [1]): I want to design a strongly typed fluent API interface to some role/permissions based authorization engine functionality. Even knowing how to shape the fluent interface so that is powerful but intuitive, as strongly typed as possible but also and maintainable, is a deep art.

One reason I know LLM can't come close to my design is this: I've written something that works (that a typical senior engineer might write), but this not enough. I have evaluated it critically (drawing on my experience with long lived software), rewritten it again to better meet the targets above, and repeated this process several times. I don't know what would make an LLM go: now that kind of works, but is this the most intuitive, well typed, and maintainable design that there could be?

1. https://news.ycombinator.com/item?id=45728183

Funny you should use role/permissions as an example here, I spent the weekend using Claude Code to rewrite my own permissions engine to a new design that uses SQL queries to solve the problem "list all of the resources that this actor can perform this action on".

My previous design required looping through all known resources asking "can actor X action Y on this?". The new design gets to generate a very complex by thoroughly tested SQL query instead.

Applying that new design and updating the hundred of related tests would have taken me weeks. I got it done in two days.

Here's a diff that captures most of the work: https://github.com/simonw/datasette/compare/e951f7e81f038e43...

Working on permissions for a large saas app, I can also confirm that the best LLM of the market have maybe a 10% success rate writing code in this area.

What % of the total amount of software (lessay lines of code or time invested) in the world is like that?

Converting an algorithm implementation from recursive to iterative: it got the concept broadly right, but was quite bad at making the logic actually match up, often refusing to fix mistakes or reverting fixes two edits later. Still a positive experience though, since it was fixable issues and reduced the amount of tedious copies I had to type

Did you have it write tests and give it the ability to iterate & validate its implementation without you in the loop?

Anything less is setting it up for failure...

Yes, but it got 99% of those then got stuck on why the others made no sense to it

It’s important to understand the tests it’s written yourself.

If you’d like some help I’d be glad to, just drop me an email.

My email’s in my profile.

Claude added a self re-calling timeout to my Typescript game loop to track time. Manually by adding 1000ms every time it's called.

I removed it and it later just added it again.

It's this small weird things where it can mess up a lot of code.

There are severe edge cases. Here are some of the last days.

Eg. Just updating bootstrap to angular bootstrap. It didn't transfer how I placed the dropdowns ( basically using dropdown-end). So everything was out of view in desktop and mobile.

It forgot the transloco I used everywhere and just used default English ( happens a lot).

Suggested code that fixed 1 bug ( expression property recursion), but now linq to SQL was broken.

Upgrade to angular 17 in a asp.net core app. I knew it used vite now. But it also required a browser folder to deploy. 20 changes down the road, I noticed something on my ui wasn't updated in dev ( fast commits for my side project, I don't build locally), it didn't deploy anything related to angular no more...

I had 2 files named ApplicationDbContext and it took the one from wrong monolith module.

It adds files in the wrong directory sometimes. Eg. Some modules were made with feature folders.

It sometimes forgets to update my ocelot gateway or updates the compressed version. ...

Note: I documented my architecture in eg. cline. But I use multiple agents to experiment with.

Tldr: it's an expert beginner programmer.

Do you have any automated tests for that project?

I'm bringing to suspect a lot of my great experiences with coding agents come from the fact that they can run tests to confirm they haven't broken anything.

The test loop is integral.

It’s kind of annoying hearing all this skepticism from people putting in the least effort into optimally using the tool. There is a learning curve. Every month I’ve gotten better results than the last because I’m constantly context building and refining, understanding how, what and when to prompt.

It’s like hearing someone say database suck but they haven’t bothered to learn about or use indexes or foreign keys.

Most of the mentioned issues wouldn't be catched by a test loop unless you have 100% automated tests (unit tests, ...)

Which isn't always plausible ( time ). The AI makes makes different mistakes than humans that are sometimes harder to catch.

It's a lot more plausible now you can get LLMs to help write those tests in the first place.

Most of these examples build ( shallow test in my LLM on the end of the task ) and produced new edge cases

Too little.

Things moved as fast as possible to migrate from .net framework to .net core 8, angular 8 to 18 and bootstrap 4.5 to 5.x

Maybe if you mentioned a more complex, lower level or niche language than typescript like maybe C, MIPS or some niche exotic systems language pushing around registers. I'd believe yu, with caveat, but with abstract high level abstract languages like Python, typescript and the likes? It's highly unlikely that you would've put together syntax in any uniquely surprising combination. Maybe yu mean yu designed a clever fix to a problem within a larger codebase so thar would mean a context/attention issue for the LLM but there's no way in hell yu wrote up a contained piece of code solving a specific problem, not tied to a larger software env, that couldn't also have been written by frontier LLMs provided yu could articulate the problem, a course-of-action and expected output/behavior. LLMs are very good at writing code in isolation, humans still have deeper intuition and we're still extremely good at doing the plug-in, wiring and big picture planning. Yu over-estimate what you've done with typescript or misunderstand what 'LLMs are good at writing code' [in isolation] means

This is a weird take. Software engineering solving and design is not about of syntax at all. Syntax can help or hinder some ways of expressing things, but the result of the design process is not clever syntax.

For example, the new shortest path algorithm that eclipses Dijkstra's is conceptual advance; it can be written in any Turing-complete language, and it's discovery had nothing to do with inventing new syntax in any specific language.

You comment betrays the literal/concrete understanding of coding that is a hallmark of novices. It's like saying as long as LLMs can write any kind of musical notation, there is no way a human can be a better composer.

I have not said an LLM cannot the same syntax or code patterns I write; I'm saying it, for instance, is poor at figuring out stuff like: How do I write types to enforce which entities and which fields and which roles are allowed for this action at compile-time? Should I use a generator, iterator, or recursive function for such and such functionality? Should this function be generic or not? How do I design my query fluent interface for the best performance? What should be the folder organization for this module that makes it intuitive to navigate and maintain? What is the best name for that function that will make it most intuitive to use? etc.

Anyone saying such concerns have anything to do with whether I'm using Typescript vs C or Haskell does not understand software engineering.

> The people saying LLM can code are hard for me to understand.

Just today, I spent an hour documenting a function that performs a set of complex scientific simulations. Defined the function input structure, the outputs, and put a bunch of references in the body to function calls it would use.

I then spent 15 minutes explaining to the free version of ChatGPT what the function needs to do both in scientific terms and in computer architecture terms (e.g. what needed to be separated out for unit tests). Then it asked me to answer ~15 questions it had (most were yes/no, it took about 5 min), then it output around 700 lines of code.

It took me about 5 minutes to get it working, since it had a few typos. It ran.

Then I spent another 15 minutes laying out all the categories of unit tests and sanity tests I wanted it to write. It produced ~1500 lines of tests. It took me half an hour to read through them all, adjusting some edge cases that didn't make sense to me and adjusting the code accordingly. And a couple cases where it was testing the right part of the code, but had made valiant but wrong guesses as to what the scientifically correct answer would be. All the tests then passed.

All in all, a little over two hours. And it ran perfectly. In contrast, writing the code and tests myself entirely by hand would have taken at least a couple of entire days.

So when you say they're good for those simple things you list and "that's about it", I couldn't disagree more. In fact, I find myself relying on them more and more for the hardest scientific and algorithmic programming, when I provide the design and the code is relatively self-contained and tests can ensure correctness. I do the thinking, it does the coding.

> Just today, I spent an hour documenting a function that performs a set of complex scientific simulations. Defined the function input structure, the outputs, and put a bunch of references in the body to function calls it would use.

So that's... math. A very well defined problem, defined very well. Any decent programmer should be able to produce working software from that, and it's great that ChatGPT was able to help you get it done much faster than you could have done it yourself. That's also the kind of project that's very well suited for unit testing, because again: math. Functions with well defined inputs, outputs, and no side-effects.

Only a tiny subset of software development projects are like that though.

> Only a tiny subset of software development projects are like that though.

Right: the majority of software development is things like "build a REST API for these three database tables" or "build a contact form with these four fields" or "write unit tests for this new function" or "update my YAML CI configuration to run this extra command".

You do know that system programming is a thing? Or that desktop applications are software too?

I said "the majority of software development". Those are both relatively niche disciplines in 2025.

Can you please explain? Are you saying all software development outside of the web is "niche"?

Not necessarily niche, but less common. Take a look at the JetBrains developer survey if you want some numbers: https://www.jetbrains.com/lp/devecosystem-2024/

I have a much more close relation with other niches than with web programming, even if web programming is part of my core skill set. I mostly interact with a few sites daily, even though I spend some time there. But I spend a lot of time with software like xterm, emacs, calibre, cmus,... and more with tooling like make, bash. While I'm not working on those, I had to become quite familiar with their working to troubleshoot some bug. Emacs is more important to me than AWS and GitHub.

Niche as in for every one systems programmer there are dozens of people writing API Glue.

By hours of work spent and lines of code produced the latter is in a whole different scale than systems programmers (which is a very badly designed term anyway).

Non web programming is not niche by any definition of the word niche.

> documenting a function that performs a set of complex scientific simulations.

The example you gave sounds like the problem is deterministic, even if composed of many moving parts. That's one way of looking at complexity.

When I talk about complex problems I'm not just talking about intricate problems. I'm talking about problems where the "problem" is design, not just implementing a design, and that is where LLMs struggle a lot.

Example, I want to design a strongly typed fluent API interface to some functionality. Even knowing how to shape the fluent interface so that is powerful, intuitive, well/strongly typed, and maintainable is a deep art.

The intuitive design constraints that I'm designing under would be hard to even explain to an LLM.

For the problems like that I consider my role to be the expert designer. I figure out a the design, then get the LLM to write the code and the tests for me.

It is a lot faster at typing than I am.

That amazing code you’ve written is a tiny proportion of code that’s needed to provide business value. Most of the code delivering business value to customers day in, day out is quite simple and can easily be LLM driven.

Agreed-I often use it when I need to brainstorm which appoarch I should take for my task, or when I need a refactor or generate a large set of mock data.

Reply on news.ycombinator.com