As a freelancer I do a bit of everything, and I’ve seen places where LLM breezes through and gets me what I want quickly, and times where using an LLM was a complete waste of time.
Building a simple marketing website? Probably don’t waste your time - an LLM will probably be faster.
Designing a new SLAM algorithm? Probably LLMs will spin around in circles helplessly. That being said, that was my experience several years ago… maybe state of the art has changed in the computer vision space.
A very simple kind of query that in my experiences causes problems to many current LLMs is:
"Write {something obscure} in the Wolfram programming language."
I wasn't able to replicate in my own testing though. Do you know if it also fails for "mathematica" code? There's much more text online about that.
My experience concerning using "Mathematica" instead of "Wolfram" in AI tasks is similar.
I was looking at trying to remember/figure out some obscure hardware communication protocol to figure out enumeration of a hardware bus on some servers. Feeding codex a few RFC URLs and other such information, plus telling it to search the internet resulted in extremely rapid progress vs. having to wade through 500 pages of technical jargon and specification documents.
I'm sure if I was extending the spec to a 3.0 version in hardware or something it would not be useful, but for someone who just needs to understand the basics to get some quick tooling stood up it was close to magic.
The question relevant for LLMs would be "how many high quality results would I get if I googled something related to this", and for DICOM the answer is "many". As long the that is the case LLMs will not have trouble answering questions about it either.
I've been impressed by how this isn't quite true. A lot of my coding life is spent in the popular languages, which the LLMs obviously excel at.
But a random dates-to-the-80s robotics language (Karel)? I unfortunately have to use it sometimes, and Claude ingested a 100s of pages long PDF manual for the language and now it's better at it than I am. It doesn't even have a compiler to test against, and still it rarely makes mistakes.
I think the trick with a lot of these LLMs is just figuring out the best techniques for using them. Fortunately a lot of people are working all the time to figure this out.
Even if your architectural idea is completely unique... a never before seen magnum opus, the building blocks are still legos.
I’d say it’s true, but the LLMs and humans don’t have the exact same definition of what “obscure” is.
Karel is almost a subset of Pascal with some keyword swaps. And there’s a LOT of Pascal (and similar languages) around.
From the PoV of a statistical based tool like an LLM, Karel is just another flavor of a very popular structure.
This is actually where I would be most reluctant to use an LLM. Your website represents your product, and you probably don’t want to give it the scent of homogenized AI slop. People can tell.
If you decide on your own brand colors and wording, there’s very little left about the code that can’t be done instantly by an LLM (at least on a marketing website).
Without playing around with it, you wouldn't know when to use an LLM and when not.
This is an industry that requires continuous learning.
Why would I do that? Well, I wanted to understand more deeply how differences in my prompting might impact the outcomes of the model. I also wanted to get generally better at writing prompts. And of course, improving at controlling context and seeing how models can go off the rails. Just by being better at understanding these patterns, I feel more confident in general at when and how to use LLMs in my daily work.
I think, in general, understanding not only that earlier models are weaker, but also _how_ they are weaker, is useful in its own right. It gives you an extra tool to use.
I will say, the biggest findings for "weaknesses" I've found are in training data. If you're keeping your libraries up-to-date, and you're using newer methods or functionality from those libraries, AI will constantly fail to identify with those new things. For example, Zod v4 came out recently and the older models absolutely fail to understand that it uses some different syntax and methods under the hood. Jest now supports `using` syntax for its spyOn method, and models just can't figure it out. Even with system prompts and telling them directly, the existing training data is just too overpowering.
For example: gemini became a lot better in a lot more tasks. How do I know? because i also have very basic benchmarks or lets say "things which haven't worked" are my benchmark.
Feels like a false dichotomy.
Have I become faster with LLMs? Yes, maybe. Is it 10x or 1000x or 10,000x? Definitely not. I think actually in the past I would have leaned more on senior developers, books, stack overflow etc. but now I can be much more independent and proactive.
LLM-based tools are a wide spectrum, and to argue that the whole spectrum is worth exploring because one sliver of it has definite utility is a bit wonky. Kind of like saying $SHITCOIN is worth investing in because $BITCOIN mooned as a speculative asset:
- I’m bullish on LLMs chat interfaces replacing StackOverflow and O’Reilly
- I could not be more bearish on Agents automating software engineering
The question isn’t if you’ve improved. It’s if the path you took to getting to your current improvement could have been shortcut with the benefit of hindsight. Given the number of dead ends we’ve traversed, the answer almost certainly is yes.
I truly believe so much of the anti-AI sentiment is the same as the Luddites.
They're often used as a meme now, but they were very real people, faced with a real and present risk to their livelihoods. They acted out of fear, but not just irrational fear.
AI is the same: it's unquestionably (to anyone evaluating it fairly) a huge boost to productivity ... and also, unquestionably, a threat to programmer jobs.
Maybe the OP is right about waiting, but to me whenever new tech is disrupting jobs, that seems like the best time to learn it. If you don't, it's not just FOMO as the author suggests ... it's failing to keep up on the skills that keep you employed.
AI is the same: it's unquestionably (to anyone evaluating it fairly) a huge boost to productivity .
It’s ridiculous double standard and poisons any reasonable discussion to assert something is a fact and anyone who disagrees is a hysterical Luddite based on no actual evidence.
To paraphrase another analogy that I enjoyed, it’s a bit like when 3d printing became a thing and hype con artists claimed that no one would buy anything anymore, you could just 3d print it.
If a stat like that is not accurately measured, it's useless.
This is all very tiring and difficult. You can be significantly better than other people at this skill.
I judge "failing to keep up" by my ability to "catch up". Right now if I search for paying courses on AI-assisted coding, I get a royal bunch for anything between 3$ to about 25$. These are distilled and converging observations by people who have had more time playing around with these toys than me. Most are less than 10 hours (usually 3 to 5). I also find countless free ones on YouTube popping up every week that can catch me up to a decent bouquet of current practices in an hour or two. They all also more or less need to be updated to relevancy after a few months (e.g. I've recently deleted my numerous bookmarks on MCP).
Don't get me wrong, LLM-assisted coding is disruptive, but when practice becomes obsolete after a few months it's not really what's keeping you employed. If after you've spent much time and effort to live near that edge, the gap that truly separates you from me in any meaningful way can be covered in a few hours to catch up, you're not really leaving me behind.
Sure, maybe crypto changed some lives, but an entire industry? I think ALL of software dev is going under a transformation and I think we're past the point of "wait it out" IMO.
Or I'm wrong, but right I'm being paid to develop a new skill professionally. Maybe the skill ends up not being useful - ok, back to writing code the old way then.
It's clearly a textbook example of survivorship bias.
In the 90s the same argument was directed at this new thing called the internet, and those who placed a bet on it being a fad ended up being forgotten by history.
It's rather obvious that this AI thing is a transformative event in world history, perhaps more critical than the advent of the internet. Take a look at traffic to established sites such as Stack Overflow to get a glimpse of the radical impact. Even in social media we started to see the dead internet theory put to practice in real time.
And coding is the lowest of low hanging fruits.
It's worth noting that SO was declining well before ChatGPT launched. It seems more likely that the decline of SO was more driven by Google ranking changes to prioritise websites that served Google ads. Certainly I remember having to go down a few results to get SO results for a while, even when the top results were just copypasta from SO.
I don't think that's it. SO was the go-to page for troubleshooting, whose traffic was not exactly originating from web search. Also, the LLM-correlated drop in traffic is also reported by search engines. Stack Overflow just so happens to be a specialized service with a very specialized audience whose demand is perfectly dominated by LLM chatbots.
Allow me to introduce you to the dot-com boom, where everyone who bet on the internet went broke.
I can be much more specific about “everyone” if you’d like.
Almost all people are "forgotten" by history.
In any case, people who were not even born yet in the 1990s are using the internet today, very successfully, so clearly you can wait.
"It's rather obvious that this AI thing is a transformative event in world history" perhaps but it's not at all obvious how it's going to shake out or which bets are sensible.
I think you are missing the point, and also the very site you're posting on.
Look at the top 50 list of most valuable companies in the world. Over half of the total market value reported today is attributed to companies which were either dotcom startups or whose growth was driven by the dotcom growth period. Dismissing the advent of the internet as anything short of revolutionary is disingenuous, no matter how many zombo.com companies failed.
LLMs have the exact same transformative impact on humanity.
But this is begging the question.
Yes, we can see that the internet was radically transformative.
But you are arguing that this somehow proves that LLMs are too, when there's wildly insufficient evidence—either on where LLMs are going in themselves, or in the comparison—to credibly make that claim.
I can't really agree. I've never seen anything from an LLM that I would consider even helpful, never mind transformative.
How are you supposed to use them?
I am conservative regarding AI driven coding but I still see tremendous value.
It makes me want to ask you: do you ever see helpful things from your colleagues at all?
No, not at all. I may be using it wrong.
I put in "write me a library that decodes network packets in <format I'm working with>" and it had no idea where to start.
What part of it is it supposed to do? I don't want to do any more typing than I have to.
Why can't you just pass any of those to an AI?
That is a heavily symbolic exercise. I will "read" the spec, but I will not pronounce it in literal audible English in my head (I'm a better reader than that.)
I write Haskell tho so maybe I'm biased. I do not have an inner narrative when programming ever.
Actually writing code is the fun and easy bit.
Where LLMs are behind humans is depth of insight. Doing anything non-trivial requires insight.
The key to effectively using LLMs is to provide the insight yourself, then let the LLM do the grunt work. Kind of like paint by numbers. In your case, I would recommend some combination of defining the API of the library you want yourself manually, thinking through how you would implement it and writing down the broad strokes of the process for the LLM, and collecting reference materials like a format spec, any docs, the code that's creating these packets, and so on.
I don't agree. It can't write code at all, it can only copy things it's already seen. But, if that is true, why can't it solve my problem?
> The key to effectively using LLMs is to provide the insight yourself, then let the LLM do the grunt work
Okay, so how do I do that? Remember, I want to do ZERO TYPING. I do not want to type a single character that is not code. I already know what I want the code to do, I just want it typed in.
I just don't think AI can ever solve a problem I have.
It may have reduced the time to an implementation, based on my experiences I sincerely doubt the veracity of applying the adjective "working".
That's the point of the blog post. If you can't even say right now whether it's for the better, then there's no reason to rush in.