What if useful AI is a fantasy?
lzon.ca
lzon.ca
edit: I had already looked up the error code the "old fashioned" way using Google to find the boiler manual, so I would have stepped in if needed, but it was literally flawless and I can't think of any time anything like that hasn't worked when I've used it for similar diagnostics.
How about the case where the "AI" misided the error code?
Did you test that? Result could be even more amazing...
> He was able to do all this instead of spending hundreds on getting a plumber out at night
... or consulting the manual.
If someone spent a lifetime mastering woodworking with hand tools, and then was shown a couple very early rudimentary power tools (lacking safety features, crude features, etc) they would rightly conclude they weren't useful. The artisan can do better work faster with less risk of dismemberment without them.
Prior to LLMs, the world's demand for good software was bottlenecked by access to competent software engineers. The people who want software just want it, they don't care about the craft. They have a different job to be done than the engineer.
An example this reminds me of is a jobs to be done theory thought exercise:
Two different first time home owners need to store yard working tools in their backyard, and determine they need a shed. The first one cares most about minimizing the time it takes to get the shed. The second one has some special constraints to deal with AND also wants to start developing their amateur construction skills. They both need sheds, but they have different values, so:
- the first one buys a shed-kit made of plastic panels that can easily be assembled in 20 minutes.
- the second one buys a power saw, power drill, tool belt, saw horses, lumber, screws, metal roofing, etc and builds a custom shed from relative scratch over a few weekends.
Another example is getting take out vs cooking the meal yourself. There are many many examples.
LLMs are already useful to many. They are also not useful to many others. To assume they aren't useful to anyone just because they aren't useful to you is a sign of absent cognitive empathy. Not acknowledging that other people have other priorities and values they are equally valid to your own.
No doubt, most of the value creation is outside of creating software. But if Nvidia and Anthropic do succeed in making better hardware and better software, then the positive reinforcement loop does seem like it could take off. And coding is a big part of that.
Maybe we don't need to understand the code at all? Hard to fathom.
Doesn't seem too crazy. This is the life of every customer (and most managers) of a software dev. To fathom it, think of yourself as the unicorn customer that can define projects, specs, and methods with competence, and even make architectural calls, with the AI being the dev.
To me, this is the incredibly clear eventuality of our profession, within the next 5-10 years. You writing code will be the same as most devs who try to hand write ASM: you will very very rarely be able to beat the compiler at its own game. And, just as now, there's a very small chance you'll go read what compiler produces.
Like in Asimov's Foundation or Liu Cixin's 'Taking Care of God'.
Maybe we need two tiers of technology. Human in the loop up to the point of AI, and then the AI layer which depends on the human stack.
Most humans are dumb, from a science perspective. The ability to be successful in science has an intelligence threshold of somewhere around 120 [2], with true innovation requiring much more (as history has shown).
Do you see any alternative to 'Taking Care of God', if we continue progressing?
But, for coding specifically, coding as an unfortunate consequence of working with computers. We've been increasing abstraction since their inception, so we can do the actual work with them that we're trying to implement with code as a means to implement it. Go ask and older programmer, and he'll say the same sort of "kids these days don't understand computers", because he had to wire up the logic on a breadboard and write machine code in a hex editor, where you use a javascript library without even knowing how to implement an adder!!!
I can imagine an organization that devotes itself to preservation of the history of technology... like monastic groups that copied texts endlessly. In a post-scarcity world, that would be a sensible outcome... people are free to devote themselves to art or science or debauchery as they please...
However, compilers have long suffered from inputs not being correct, so we have put a lot of effort into finding ways to validate that the input is correct relative to the output. Which turns out works just as well for LLMs, to the point that validating that the input is correct relative to the output ends up offering the same assurances for both compilers and LLMs alike. Compiler output correctness alone is not actually all that beneficial in practice because, again, it is only as useful as the correctness of the input and inputs are not trustworthy without "double entry accounting" to go along with it. Once you have "double entry accounting" in place then you already have what you need to turn LLM correctness into being binary as well.
More useful for drying clothes than a blender would be though. Or an LLM for that matter. No real point here I just thought it was an interesting example you used because it's still something that does work if you're lacking better options
> Another example is getting take out vs cooking the meal yourself
Interesting analogy
Buying ingredients and Cooking for yourself is almost guaranteed to be cheaper and healthier for your body in the long run. It's pretty difficult to eat out cheaply and very difficult to eat out healthily.
I dunno, might turn out to be some parallels there. I suspect that much like eating too much takeout, LLM usage makes us less fit.
So far it seems that we can't all be good/capable at everything.
Maybe you write all your own code and cook your own meals, but do you do your own taxes? would you defend yourself in court? would you fix your car yourself?
Some people may answer yes to all the above, but most people outsource most things. Many people can't afford to outsource to other people, but can afford to outsource to LLMs.
Again it comes down to the job to be done:
- does the person want to develop or practice a skill, or do they just want it done? What is their budget, and how important is it to them to minimize certain risks?
I'm probably biased, but I think when it comes to professional work it should always, 100% of the time be the former. People who "just want it done" are the reason every piece of commercial software is becoming dogshit
> So far it seems that we can't all be good/capable at everything
Sure, but then stay the fuck out of the area I'm good at.
That's the whole thing. I earn money by building software. I spend money at a restaurant to buy food from people who are good at cooking.
If people who are good at cooking use AI to build their websites, then people like me don't have jobs, don't have money to buy food from them anymore. Specializing is how society functions. People no longer need my specialty thanks to AI? I can't get food anymore?
Kind of a shitty deal for me!
For all of history the environment has put evolutionary pressure on living things. If you choose to make a deal with the universe that you get what you want without having to adapt to changing circumstances then you're going to get a shitty deal. If you embrace what's outside your control and adapt gracefully you will get a good deal.
What that means in this particular case with AI and your skills as an engineer I can't say. I don't mean anything specific by it. My point is more so that a big source of unnecessary suffering is wanting the universe to be something that it's not.
Thanks. I'll change my attitude about it so that when I'm inevitably starving on the streets I can look back and think "thank god I didn't suffer unnecessarily before when I was worried about this outcome"
I mean that if you accept that the market demand for software development is evolving and you evolve with it you won't end up starving on the streets.
If you assume it's inevitable that you will become useless and starve then it's a lot more likely to happen.
There is a concept in racing "don't stare at the wall". It means when taking a corner at top speeds of you focus on what you're afraid you will crash into you're more likely to crash into it. So instead the idea is to look at where you want to go (around the corner).
So what I'm saying is that if you accept the universe for what it is and focus on what kind of life you want to have and what you need to do to have it you're more likely to get it.
It seems like there is unlimited demand for software, and software engineers who maintain skills relevant to the current market will find work.
Literally nobody thought this.
AI is not going to go away and it's not going to stop improving. That would go against the entire history of computing.
We have levels of improvements in the R&D pipeline in every area: hardware, software, model architecture and training. The models will get larger, architecture more sophisticated, computation dramatically more efficient. New materials, paradigms, more efficient nano-devices, scaling up manufacturing for better devices that are already out of the lab, etc. are in progress pointed towards multiple orders of magnitude efficiency gains. And by the way, that is not at all unusual -- we have been making large and small innovations in computing efficiency for decades.
There are still some things lacking in AI -- it still is jagged intelligence. Give it a few years, people will be nostalgic about the time when humans could still point to some victories here and there. That doesn't require any major breakthroughs -- just continuing to increase the size of the models and improving the training.
If we consider (and I’m not saying this is true, just for arguments sake) that this agency is important as other humans enjoy talking to humans about a goal, rather than machines, and we think this will never change.
In this peculiar case, it actually doesn’t matter how “good” llms get - they will never cross a barrier fundamental to human nature?
Also, they are really VLMs and VLAs and there are other types of architecture and there will be many more.
So if you think that there is something salient about humans that can't be reproduced eventually by AI, then look into religious dualism.
Contradicts
> not what I was discussing. I was talking about programming.
To be concrete: my argument is that to
> "simulate humans or reproduce all of human nature"
Is exactly what would be required for an LLM to replace a human as an effective interface for programming. Crazy hill to die on, I know, but that's the goal.
I am purely talking about programming. The reasoning looks something like this.
1. LLMs are undeniably useful as assistants when programming.
2. At some point though, they are leveraged too much. This leads to humans having an inability to reason about the generated code.
3. The LLM now needs to be as good of an interface as a human being to be of any use.
I think most people would agree with this distinction. Where people argue is where the cutoff point lies - and it's actually hard to find. How do you know when you have transitioned between the three stages? It's tempting to say, if the speedup is marginal, it isn't worth the risk.
In a broader sense, computational chemistry and drug discovery is already exploding- very, very hard to see that genie being put in the bottle. They've probably created such a large backlog of potentially useful outputs that it would take centuries for humans to sift through.
Useful AI is already here! Get over it. Your term "jagged intelligence" is very good, I will have to write that one down. Jagged indeed.
I find it baffling that some people had to find this out the hard way. You already knew that it's — if not more work — then at least more tedious to study existing code than write your own.
People have been choosing greenfield rewrites over grokking legacy code since forever.
Practically speaking, is there really a difference between prompting an LLM to create a CMS for a website and installing something like WordPress or Ghost? The setup requires no understanding of the underlying software, but you'll have to build up a mental model from scratch if you want to extend or modify it later.
Of course, it's not going to please anyone that likes the process of programming or knows enough to tell that the output is mediocre at best. But it's very useful for people that don't know how something works and want a 'workable' solution that functionally does what they need. It's basically the next iteration of the WYSIWYG web design tool, the CMS, the site builder, etc. A professional software engineer would be horrified by the code they output, but a non-technical manager type wouldn't notice or care.
There is some difference; when you install WordPress or Ghost, your problems are shared. There's a community that goes along with the software and a "shared understanding in the world" of how it works and where the rough edges and limitations are. A lot of the time, LLMs will actually be better at modifying WordPress or Ghost to do what you want than they will be at fixing the CMS you built last Tuesday.
When you custom build, you can get an exact fit for your needs but your misery is yours alone.
Yes. Like working in a team. It can be hard to work on a team and to have to understand what your colleagues did, and how to fix or extend it. What if working in teams is a fantasy?
And lacking in self awareness to publicize that.
Obviously some people are using it recklessly, it's the same with everything.
Do you think it's more likely that 'the entire world is wrong and you are right' ... OR ... that you just personally have not figured out how to use the technology in way that makes sense and/or you have a bad sense of of what quality means in most contexts.
AI produces easily 'better than human' code where it has a lot of guardrails, and is probably 10x better than humans at hunting down and fixing bugs.
Software is 90% 'maintenance' , and a well maintained code base provide ample basis for the AI to understand what design and formatting to use for a given solution.
It definitely falls down on it's own architecture, it's not that hard to work along with the AI.
It doesn't matter what we think of Altman or whoever, or whatever or AGI or 'the bubble' - that's all noise. It's literally the most blatantly obvious 'useful' thing to happen to tech since the advent of machines - it takes a 'collapse of intellectual contextualization' to not realize that.
You're watching cars replace horses, they are not the same thing.
Yeah, see, I've seen this movie before. I've seen it with Lisp. "You just haven't used it enough to reach enlightenment! We all have! Keep going; you'll see it too!"
And then with Haskell. Exact same looking down on everyone who doesn't agree as unenlightened, as people who Just Don't Get It.
Both Lisp and Haskell are useful languages. Neither is the future of mainstream programming, not just because the mainstream is unenlightened, but because those languages don't fit the way most programmers think.
People are different. Different languages fit different people. If Lisp fits how you think, when you find it after wandering through procedural languages, it feels incredibly freeing. But if it doesn't fit...
So I get very nervous about "you just haven't figured out how to use it" claims. Maybe it's true. Or maybe it just doesn't fit a bunch of people, and browbeating them with claims that they're unskillful does not make it fit them any better.
'Lisp' is an arcane tool and it's highly debatable in the objective sense if 'Lisp' is better.
The analogy is someone is saying that 'cars are useless' and that everyone should just keep riding horses.
I'm saying that we can debate the merits of cars, but it's absurd to suggest that they are not useful, and it's reasonable to conclude that someone who cannot 'find value' in cars, has not learned to drive them.
Then debate the merits.
Take a complex, sophisticated codebase you wrote, that only you have deep understanding of, and throw Codex or Claude at it and ask it questions. How does the Foo work? What happens if the user tries to Frozzle a Bar? Give me a guided tour of how the Baz feature was implemented, top to bottom of the stack.
Personally, I was amazed at how accurate the results were, on a codebase where I knew the answers already. Try to pretend that's not a useful tool to use when you have to fix bugs or add features to a large codebase you aren't already familiar with.
how will I know the answers are accurate?
It's similar to how Google has been useful for the past few decades despite the internet being full of junk info. Except Google could only provide semi-reliable information about stuff that's public knowledge with lots of people talking about it, whereas Claude can provide semi-reliable information about your own code that nobody else has ever laid eyes on.
A few months ago I used this to re-acquaint myself with code I myself had written back in 2017-2018[1]. I wanted to add some new features, and I did NOT want AI to write them, because the existing codebase was my own sweat and tears and running it through the slop extruder would cheapen that original work. I wanted to write all the code myself with full understanding of what was going on. Now, it had been many years and the codebase did some pretty esoteric stuff. I was the only person in the world who ever knew what that code did, and I didn't know it like the back of my hand anymore. Particularly the stuff with runtime CIL generation for result set processing.
I did a lot of reading my own code and a lot of thinking about it to get that understanding back, but it was a very helpful tool to be able to bounce my questions off an agent during the process and then investigate its answers for accuracy. And greater than 90% of the time its answers turned out to be correct.
Honestly, this reads a bit like you have not tried it, because the answer is 'kind of clear' if very nuanced.
You get a sense pretty quickly for the 'boundaries' of the tech - and though it will make errors for sure - you can sense what kinds of errors it's inclined to make.
Why? Unless you foolishly choose to target something like Javascript, or Python, or the like where the training corpus was full of poor quality code then the LLMs stay pretty true to how you'd write the code yourself. LLMs are only as good as what they were trained on, but luckily there was already a strong movement towards "all codebases should be the same" before LLMs, and as long as you hone in on those ecosystems then LLMs trained on those ecosystems can work quite well at producing quality code without the need for heavy oversight.
It depends on how much 'rails' there are.
More rails, then there is a lot less to examine.
You might find it to be an interesting read, but the software will be fine if you don't.
" You already know that the quality will be on par with what you would have written yourself."
No - I totally don't agree.
I've almost never seen a situation where one does not have to read, other than for bug fixes.
In almost every case where I have 'gone under the good' to AI created system - it was bad.
The AI is terrible at grasping and coding concepts. It names things poorly, and over iterations, it will turn 'state' into a hairball.
Try asking it to do an DOM event adapter to filter and map events, like doing 'ime end even guaranted' - it will do write something and write tests that all pass, but there will be holes.
The logic will be inconsistent and won't follow an hard spec, and even getting it to write a hard, clean spec won't work.
Let me put it differently: it's not the 'code' that's the problem - it's the concepts that it is trying to represent, which it screws up.
Only where those 'concepts' are already clean, such as in a really good code-base that you only need to tweak - does it work well. And amazingly well.
But for doing anything from scratch - you have to read the code.
I don't think it's going to get better any time soon because I think there is a 'noisy channel' problem here, in that it can only get so good without proper specification, and specification is 'the developers job' that cannot be replaced, at best we can get tools to support us there.
Why? While you can technically build a DOM event system in any language, you wouldn't, so this sounds like you are talking about Javascript (or Typescript, but same thing)... Which we already know LLMs cannot handle because the training data was full of poor quality code and thus LLMs also produce poor quality code. So, yes, you have to watch Javascript like a hawk here because the LLMs will go to crazy town if you don't pay attention. But that was already acknowledged.
Perhaps you accidentally chose the wrong example?
Second - it is very wrong to imply 'the AI has bad training data for JS / TS' - that's generally not how it works.
Third - the main point is not the code, it's information distillation as I suggested. It has nothing to do with the programming language. Yes - it might plausibly work better in some than others, but that's not the crux of the issue.
As to to your question the 'DOM event filtering' is just a good example because there are ambiguities in the problem space.
The spec is odd, browsers have different stated behaviours, different actual behaviours - this creates difficult problem space surface. It doesn't matter what 'la
It is an example, but an example that doesn't seem to fit the discussion.
> it is very wrong to imply 'the AI has bad training data for JS / TS' - that's generally not how it works.
Where would it have found good training data? The popularity of JS/TS means that the vast majority of the code that was written for it prior to LLMs is pretty lousy. Worse, even among the codebases that could be considered good, there is very little consistency between them so there is real good way for an AI to converge on what a good codebase even looks like in that ecosystem.
Regardless, no matter what explanation you want to choose, it remains abundantly clear that LLMs show time and time again that they struggle with JS/TS where there is no such struggle in certain other ecosystems. If you want to leverage LLMs to their full potential you are simply not going to be able to choose JS/TS (among others).
> As to to your question the 'DOM event filtering' is just a good example because there are ambiguities in the problem space.
Perhaps the disconnect here is that you have forgotten, but the discussion was about code quality. What about DOM event filtering in particular would lead an LLM that otherwise produces quality code to suddenly not? I have seen nothing to suggest that, in ecosystems that LLMs are suited for, that code quality declines when the problem space becomes more difficult. Code quality and problem space difficulty are orthogonal.
Or perhaps I am misunderstanding you, but my read here is that you point out that when a problem isn't well constrained an LLM may provide a solution that doesn't meet your expectations. That is true, but code quality and meeting user expectations are not one and the same. In fact, in the startup realm you often hear things like "users don't care about your code quality, just release something!" to really drive home that they are in entirely different dimensions.
I think one of the issues with LLMs that confuses people is that they're such a broad open-ended tool that people don't know what to do with them and so conclude they're not sure if they are useful, but in a meta way you could just keep asking the AI questions like "how do I make good use of LLMs in this case" and so on
They seem like an upgraded kind of search engine for text prompts, if nothing else
If you don't have a useful mental model of the architecture that's a communication / review / documentation problem, not a problem with the entire idea of codegen.
Even if the worst happens in the AI market some uses will still remain. Far fewer, I suspect, than will be demand the computational capacity of all the DCs popping up everywhere.
it usually helps if you know how something is to be done, you just want to save time not doing it yourself. this way you can give the LLM intermediate structs and transitions to work with, and constrain it this way not to do something stupid.
It's like saying a calculator can do your taxes.
Doesn't work that way.
But grifters gonna grift
It's a psychotic delusion.
Useful LLM is not.