Hacker News Reader

Comment by simonw | Hacker News Reader

Something that's been amusing me about Cursor is that I feel a lot of the excitement about it isn't so much about Cursor, it's people realizing that LLMs have got REALLY good at writing code now. GPT-4 was good for the past year, but the latest models (especially Claude 3.5 Sonnet) are spectacular.

Those of us who've been copying and pasting LLM-generated code back and forth from ChatGPT and Claude.ai for the past year had figured this out already, so Cursor wasn't a huge surprise for us.

For a lot of people Cursor is the first time they've really appreciated how good this stuff has got - so it's getting a massive amount of buzz, much of which should really be credited to Claude 3.5 Sonnet.

(Cursor have done an excellent job designing editor features on top of that model and deserve credit for that, I'm just a bit tickled at some of the excitement which basically boils down to "Huh, LLMs can write good code now".)

View on HN

I don’t share the sentiment that it’s “very good”. It’s strongly mediocre at basic tasks and regularly fails at anything complex.

Have we been using the same models?

Claude 3.5 Sonnet writes me excellent code in a variety of different languages. I have a bunch of examples here: https://simonwillison.net/tags/claude-3-5-sonnet/

Yes we have, but probably for different problems.

For Python it's bliss. For non-std C++ it's hit-and-miss. For CUDA C++ it's terrible.

For POSIX sh it generates kind of ugly and unnecessarily complex scripts.

"POSIX sh it" hehehe - my inner 5 year old is too pleased right now.

That's a failure of imagination. If I can architect and describe my interfaces and have Claude write the current component, then I can repeat with every other component and write minimal code / corrections.

At the same time, I've minimized complexity in my codebase. It's a win-win, and honestly probably the next paradigm of software engineering.

Depends on the problem at hand. Some code has high essential complexity. Claude doesn’t do well with math heavy work in my experience

I'm getting what I consider very good results. Yes, I have to iterate it and talk to it, spec very clearly, but if you know pretty precisely what you want I've found it very reliable.

I usually try to actually provide AWS resource IDs / VPC id's, bucket names and everything and at least with AWS / Python stuff it is good (yes, some corners still but the trend is very good).

My prompts aren't short but it spell out very clearly what I want. To process 2 million XML files let's say stored in S3, using an SQS queue ZZZ that will pass in S3 object ID's in this format, store results back to RDS Postgresql here. Use Loguru or whatever for logging. I provide sample XML files, sample results, schema for database (DDL) etc etc. Then you can iterate on this - deploy using docker to ECS, use fargate, then do some scale out when queue depth is deep, go to asyncio or multi-process or multi-threading etc etc and just build. It goes quick.

That doesn’t sound like an inherently complex programming task - just simple mindless munging

I must be living in a different universe, every single LLM I've tried are shit at the details/nuances of writing code.

Claude has no trouble with SQL queries, writing tests, utility functions, component sketches (React). But as soon as I start giving it reasonably complex stuff like interdependent generics (TypeScript) or intricate custom logic implementation on top of well-known frameworks, it quite often goes around in circles or outputs code with incorrect syntax.

You know, the stuff that I would actually feel as though I was receiving intellectual help with.

Current LLMs are _great_ at doing chores. They are not so great at producing cohesive, professional-grade code.

I tried getting Claude to help me model an event-based store with moderate type safety and it was a total disaster. It was randomly trying to convince me to turn it into a state machine using a library in Python (I was working in TypeScript).

This is so far from awesome as far as productivity goes. I do find cursor suggestions useful here and there, but it’s typically filling in the simpler details of an implementation I had to design without its help. Ultimately it doesn’t save nearly as much time as I hoped it might a couple years ago.

Still, cool development in terms of dev tools and I’ll keep experimenting.

LLMs...60 percent of the time it works every time.

>>I tried getting Claude to help me model an event-based store with moderate type safety and it was a total disaster.

That's just not how you work with a LLM. You don't give LLMs problems to solve. You do the thinking part and solve problems, and ask the LLM to implement small blocks of code, like the smallest possible, and you incrementally go from there.

This really is some what like asking the whole question in the Google search bar, not seeing any result and saying Google search doesn't work.

>>it’s typically filling in the simpler details of an implementation I had to design without its help.

Most complex solutions are just reusing and connecting simpler solutions.

> ...and ask the LLM to implement small blocks of code, like the smallest possible, and you incrementally go from there.

So it's autocomplete++? Talk about an insane level of hype for an incremental improvement to what we already have...if that's actually what we can hope to expect from these LLMs.

At their heart LLMs are basically a mechanism to guess(predict based on probability) the next word based on what it has already done/seen so far. How it goes about guessing them, or how it gets the context is based on attention and multihead attention functions. You could say that these functions provide which part(word) of a question/sentence must get a higher weight. That basically provides context based on which it predicts what needs to come next. How it predicts is basically your plain old neural network. Just like how in linear regression you know, if your model is predicting a straight line, you know in which area the next points are likely to appear. Similar mechanisms are used here to guess what the next words are likely to be.

It is a extreme auto complete feature in the context of code for sure. Note, LLMs are not sentient. That means they can't be held responsible for making decisions. Even more so code decisions.

Now you can argue its nothing special. But its some what like arguing eclipse/intellij are not special when compared to vim/emacs. That is just splitting hairs. IDE's definitely do a lot of productive work compared to plain text editors.

The initial demo's on LLMs confuse new users a lot. The demos go on the lines of giving a sentence like 'Implement a todo list app' or something like that and LLM writes some code implementing it. That's a wrong way to work with LLMs. Don't outsource your thinking or give it whole blanket problem statement to solve it. Think about LLMs like tools that do a lot of quick text writing for you you given the smallest, non ambiguous, atomically implementable/rollback statement possible.

It gets a while to get used to this, but once you are, you are more productive.

This is how I was using it. I was trying to work through each part of the model, but kept running into an issue in which it would recommend implementing things with Python instead. It wasn’t impossible or even inadvisable to do it with typescript, so I have no idea where the impetus to switch to Python came from.

The issue occurred specifically when I was pointing out that type inference wouldn’t work as expected for typing the store and its corresponding events. I’d feed it some types that were close to what we needed and it would give up and show me how to do it with Python. I gave up after (I wish I was exaggerating) around 2 hours of experimenting with convincing it to believe it was possible to build and type the store correctly.

The store is built and working properly at this point so who knows, maybe the meta layer of typing things really throws LLMs.

Which language are you using, and what kind of code are you writing? So far I've had good results for Javascript/Typescript and SQL for web apps and Unreal C++ for game development. I can imagine it would do way worse though if you were writing something with a lot of domain-specific knowledge, really cutting-edge stuff or in a more obscure language like Fortran - but then again, I think that most developers would also struggle more in those conditions.

Part of what I find useful about LLMs is that it's almost like rubber duck debugging, except that the duck talks back. You have to explain to the LLM exactly what it is that you want it to do to get the right result, then it gives you a solution. Mostly the solution is right, but when it's wrong you'll better understand the problem through the act of writing it out.

> You have to explain to the LLM exactly what it is that you want it to do to get the right result, then it gives you a solution. Mostly the solution is right, but when it's wrong you'll better understand the problem through the act of writing it out.

The solution is either correct or incorrect. In general its level of incorrectness is not trivial. My language of choice is C#. Anytime I ask it to generate a series of classes (say a data model of a particular domain, using specific well-known patterns), it gets it wrong, quite wrong and it would take more effort to explain it to unfuck itself, by which time it would have forgotten 3 deliberate instructions from 5 interactions ago. It’s a brilliant moron.

>>The solution is either correct or incorrect.

Not even humans write 100% correct solutions. Everyday programming is not a theorem proving exercise. Programs are correct enough, to cover a agreeable amount of test cases.

>>Anytime I ask it to generate a series of classes

Generating a class is too big and broad of question to ask an LLM to generate. My chunks of inputs and outputs to/from an LLM are often on the lines of a single for loop, if block or a small chunk of code that can't be written in more than one way. There are several steps from here to a complete class.

LLM is your classic socratic thinking tool. You really to have to learn to ask and work in chunks of questions that are smallest, and involve only one change from the previous question, and are atomic enough to do a easy roll back. And you need to build from here.

As experience with the series of books The Little Schemer shows, this unfortunately is not everyone's cup of tea. A lot of people struggle to keep a train of thought in their minds and work in that workflow.

To that extent I see LLM are not for everyone. Just like how Google search was great and helped a lot of people, but the majority just couldn't bring themselves to sit in front of a screen and work for hours stitching solutions from various places on internet into a working solution.

80% of folks are doing web stuff, as soon as you deviate from that the quality must go from 9/10 to 3/10 or whatever

Also have you tried shorter blocks ? "Write a function like this" instead of "write the app"

I would love to see someone on stream use AI really effectively.

I really try, but for the life of me, I can’t get AI to generate anything useful beyond simple tests.

They’re still autocomplete++ for me.

Autocomplete++ is the killer feature for me, especially for more tedious thingsike SQL column naming.

Agreed! But also...

Rather than your seeing someone stream "really effective 99th percentile AI coding", maybe it would be more illustrative for us all to see someone stream the "50th percentile developer unassisted" experience.

I suspect that's the thing we're all missing to help us understand the potential value of AI coding: how slow and ineffectively the median software developer works, bc they just quietly slog away and don't talk much about how they work :)

I humbly include myself on that thought btw

I also don’t understand the appeal. I never wrote boilerplate code in my life.

I recently spent the whole day debugging open source third parties on multiple OSes while using Wireshark to understand how the whole thing worked. Then I typed 5 lines for a fix in less than 30 seconds. How would an LLM help me here?

> I never wrote boilerplate code in my life.

You either have not written very much code, don’t know what boilerplate means, or are intentionally operating under an overly broad definition

You’re right. I don’t write a lot of code. It’s always very specific C++ stuff that does not need to be repeated a hundred times, or if it is it can be a virtual method declaration that I copy in a subclass. No need for a LLM for that.

I’ve only seen boilerplate used by web developers, but I’m not doing that.

You shouldn’t speak so confidently and condescendingly with so little experience.

Graphics programming, data science, anything involving handling user input and probably many more classes of software require ceremony and boilerplate to get going.

Vulkan famously takes 1000 lines of C++ boilerplate before you can draw a triangle.

Look into writing a X or Wayland compositor and you’ll see a mountain of boilerplate involved in handling input loops, displays, and communicating with the kernel.

Please don’t misquote me.

I want to see how someone who’s confident in using these tools write code with them.

I’m not as interested in the median developer’s coding practices.

I find it effective to treat it like a junior, and provide detailed and specific tasks. Ie usual LLM usage.

A recent e.g. "Use the Javascript module pattern to create a module that stores audio chunks, recording state, and exposes functions to start and stop recording, and to store audio data"

It's a really simple task, but that prompt saved me 20 minutes of typing and futzing about. It also named the module sensibly, based on context from the rest of the code I guess. Then it prompted me on what to change in the calling code to use the new module (this wasn't 100% right, but still saved me a couple rounds of running the code and fixing problems).

It's taken me a while to change my habit of working out how to do what I want to do and then editing code, to rather just saying what I want to do, and being specific about it.

Even in cases like that, I find that it’s often wrong about whatever package I’m using, unless that package has had a stable long lived api.

I can maybe pull snippets out of AI generated code, but more often than not I just ask it for quick examples. Like I would do for stack overflow.

A good stream example might be this on a podcast, its got a few cuts but generally can get the point and its by a complete beginner, 1 month in

https://www.youtube.com/watch?v=ehK-QqPstJ4

Cursor wow'ed me a year ago when I first used it. Been using it ever since. People are only starting to notice it now because of the virality machine. AI has been reasonably "good" at code for awhile. I think Karpathy finally talking positively about it a few weeks ago is what started off the current hype cycle.

How would you compare GPT-4o with Claude 3.5 Sonnet?

Not parent but Claude any day of the week, it seems to keep context and do what I ask it better.

I've gave up on GPT-4o a while back, I've spent the last 1/2 hour (more like wasted it) trying to get Claude 3.5 Sonnet to do my bidding (nothing super complicated) and it's a complete mess. It does forget plenty. It gets things wrong. It's like the weather man, you just can't trust a word he says.

For my prompting style and the languages I work with - Python, JavaScript, SQL - I think Claude 3.5 Sonnet is slightly better.

Thanks! It's interesting how this perception seems to vary based on language.

Reply on news.ycombinator.com