The Normalization of Inexplicable Failures
ihatethefuture.com
ihatethefuture.com
I am also big on testing (the correct things). And nine-nines (big on Elixir).
And... I'm also big on agent-assisted dev. Which requires pretty much every check in the book to stay productive in. And that's fine to me. I've seen bugs that I wouldn't have made myself. And I've also seen my own bugs fixed. They've all gotten fixed in short order. I don't see why this is a problem.
Raise your personal standards.
Thing is, the unreliable-software situation was already untenable before agents (in poor hands) made it worse.
But it’s pretty clear that most people are not. For whatever reasons (mgmt pressure, trying to get ahead, skill issues, etc) they half ass it, accept the 10% (silent) fail rate and blame the bad outcomes on the AI as if that absolves them. Or, adopt the attitude that 10% fail is fine, and people who say otherwise are being picky, or are anti-ai luddites or whatever. You should accept that things will suck.
Concepts like good design, security, quality are abstract and hard to measure.
Time, cost, revenue etc are easy to measure. The “quality” people eventually get push aside by the money people.
This is not new, but LLMs further tilt the balance
You speak as if llms had their own minds. Every time anyone talks about AI doing this or that they further reinforce this idea that there isn't a person behind all this. There's always someone watching.
With that said, when you say that llms tilt the balance, who specifically do you say that's driving llms to do that?
It was dreadful, but the volume was a single-digit percentage of what you see now.
Even those develoeprs who made a living opying from SO *still needed to make that code work for their system!"
I don't think this has ever, as a rule, been true.
In the past it wasn't really common at all to add 10 features in an evening without actually trying to use those 10 features by hand yourself.
Personally I think it's wonderful that tools for these spaces exist when they didn't use to. But it's also ludicrous to say that anyone with Claude can replace even mediocre homemmade stuff in any dimension other than "being worth building even a bad tool" or "getting to semi-usable faster." Currently you still benefit massively from knowing what's going on behind the scenes, and from knowing "software engineering 201" type stuff around what sort of testing would be helpful where, vs accepting model-default-output everywhere.
I will take time to address you comment fully, there are lots to unpack, but I'd like to stop here and point out that comparing "absolutely some stuff" from today to the "average 2018 project" might be unfair. If you are going to compare the bottom of the pile of synthetic code, you should do the same with organic code of back then lest you draw an unfair comparison.
Make it the bottom 70% AI vs. the bottom 10% human if you want to. You're still going to be well within a sea of AI-slop because the volume is that large. The big thing about bad human code is that it tends to still be compact; I can read in some minutes the intention and what does(n't) work.
For each vibe-coded project I gotta do a tiny expedition just to get the basic idea of what's going on. let alone figuring out if things actually work as intended.
I envy the environment you've worked before, because you definitely had a better experience than mine. My first job was working on a product -- not a prototype you see, an actual product with paying customers generating over 100k USD monthly for the company -- that was completely designed by interns from the ground up. Once we received a pull request on a part of the system that dealt with calculation reports that made snapshots of the relational db into MongoDB. This application was in PHP using an outdated CakePHP framework -- which was outdated for a good reason since it used to introduce breaking changes in minor releases (citation needed, but who's got the time these days...) -- and the merge request that once-employee left us to figure out how to merge was a single 400 lines function. You can extrapolate from it to infer the quality of the snapshots we were taking and you probably wouldn't land too far from the horrors we've seen.
So pardon me, but my experience shocks violently with your affirmation.I think you underestimate what the bottom 10% really is.
I don't condone this view, but i understand where they are coming from!
Yes but that's the big thing, now isn't it? These are nice tools, used wisely. But their unwise use, oh boy...
The problem is one needs to be in a situation where the incentive is towards quality rather than speed. But that situation rather rare now - thirty years ago, Microsoft won the office wars with crap that had features. And nothing has fundamentally changed in web development since the LPad crisis.
The problem is those companies whose incentive is to allow bugs where it's the involuntary users who suffer will bite you no matter what quality you make your own software.
I have definitely seen more bafflingly-poor OSS software that just plain doesn't work frequently now than before.
But it's mostly software that wouldn't have existed before because it's trying to do super-niche things. So on the "hobby" side of things, whatever.
But from a "trying to develop software as a business that you want to be a going concern," quality from people who should know better is less tenable than it used to be.
That may be tolerable for some user-facing app. But what if we start normalizing failures in the libraries, the infrastructure, and the compilers? Everything descends into a mess of unreliability, and that slows EVERYTHING and EVERYONE down.
Everybody wants to be the ones building cheap slop, hoping somebody else makes protections and repairs...
There's definitely still AI/LLM integration happening, but is kept out of specific areas of the business.
The normalisation of inexplicable failures is, in my opinion, the consequence of programmers being unable to refuse their managers who have fallen for the LLM siren call, lest they lose their jobs.
If you randomly screw up customer orders (think of DoorDash or an online shop or Airbnb). They lose trust in you and you lose your business to the competition. Going happy go lucky and being irresponsible in the business can bankrupt most* businesses.
* well, of course except the criminal empires which are bailed out by our tax money.
So that is a strange choice for repeatable, understandable operations. Might as well use Jev.
Firstly, the transaction failed and notified you about the error - that's certainly intentional.
Second, there's failures that would be invisible to you as the customer, such as "instead moving $100 from account A to account B it credited account B but didn't debit account A, without generating an error". Those are mostly the systems I'm talking about being insulated from AI development. Without more detail about the exact problem you had it's hard to tell if it's a failure in customer facing systems or backend infra.
Third, I'm assuming you were able to reach out to the bank directly and resolve the issue by talking to a person (if the issue was really outside the norm), which isn't something you can assume will be possible with a lot of customer service ops these days (or you're going to be waiting hours/days for that callback).
Fourth, you can't really know the error rate of the bank's systems, or how common a given particular error is. It may be a known issue, or it may be a completely unreported one. Assuming it's an error with something on the backend/backbone of the bank's operations, it's running code that can be inspected, reviewed, understood, and fixed - sometimes by a very expensive COBOL consultant.
Perhaps designing with assumptions that the entire stack is imperfect is the route to higher quality, along the lines of chaos monkey, which randomly kills processes on production boxes… which ensures resilience is real.
Even without AI our tech stack is so deep it’s hard to imagine every feature at every layer being provably correct.
The point is that it boils down to writing the tests correctly, regardless of who is implementing the actual code. Hand-written code without test coverage has the same problems as AI generated code.
https://ravimohan.blogspot.com/2007/04/learning-from-sudoku-...
Or have them rig the tests so that they always pass.
Not unlike legacy code, where the original developers are long gone and no one knows when that system breaking bug is about to wake up.
Even in the best of cases tho, fully human written and reviewed code will still fail eventually.
So I would say we are not normalizing failures (yet) but rather normalizing legacy.
I know because I've spent the better part of a year having them do exactly this as a precursor to rewrites in more perfomant tech stacks. Once this research is done, it's a fairly safe and mechanical translation job.
Say that with confidence now. You'll say with shock and incredulity later, "It can be fix? Right?!?!" (Cue some meme). Reproducing bugs is often hard for well designed systems. For randomly designed systems, it's a nightmare.
Fixing software is hard and fixing bad software is harder and fixing software where you didn't even think at the time how to make it maintainable is hardest.
Legacy systems are usually systems that were well designed at the time. That's why it's worth fixing them now.
--> And none of this is saying AI designed systems are bad. But if the only criteria is "code is deterministic by default, it works and we can fix problems later" then AI seems likely a terribleness accelerator.
Code written with AI can be fixed, if well designed, they are not mutually exclusive.
Also legacy systems were absolutely not always well designed, I've picked up a fair share of systems that were absolutely not well thought out, usually due to time pressure, or any other context specific reasons.
I do believe we are creating legacy systems faster now, and it's because teams are unable to keep up with the speed of changes, not because their system is not meant to be maintainable.
My point remains tho, using AI doesn't have to mean normalizing failures.
Nice way of putting it.
Compilers actually actively try to stay deterministic in their output.
The main argument for LLM-driven development is much simpler: "It will get better".
The current state of LLM coding is about a year old. Imagine if we dismissed human coding efforts after a year. Rust, Python2 -> Python3 transition, Python type checking, Windows, C++, … nothing of that was done in a year and emerged in perfection in the first year. Everything takes ages to mature into a usable product. LLM coding is still in the "throw mud at the wall and see what sticks" stage, give it some more years and see how it will develop and what approaches actually work at. For the time being, LLMs are just the most useful development tool in the history of development tools, that's a pretty solid start in such a short time.
Manufacturing lines have tight tolerances. Science has 95% confidence intervals (or greater). HFT has fractional pennies to steamroll up. But “business” (broadly), leadership, macro decisions 3+ steps removed from the coal face can safely operate at wider tolerances.
I cringe whenever I see “xx.xx% growth” on a report as if the value in that hundredth of a percent place is going to sway anyone’s opinion one way or the other. It’s superfluous, wasteful and I would argue, harmful.
The U.S. Marines teach the “70% solution” which says that making a decision that is 70% correct now is better than making a 100% correct decision later.
The speed of your OODA loops is critically important, and cannot be overlooked or expensed in favor of determinism, predictability etc for its own sake. (After all “no plan survives first contact”)
> For many users, however, the actual experience is roughly just "stupid thing sucks." Software already feels capricious; more failures just change the rate of frustration.
I am betting author does not use cloud services much. It is not just "users", it's developers as well. Github is returning 5xx? AWS service does not work? Your email did not get delivered? Nothing we (developers) can do, "stupid thing sucks".
One can be simultaneously a developer and a user. Distributed systems [0] weren't invented five years ago, after all. ;)
"Github owns this part that we rely on for correct operation and we can do fuckall about it when it fails." is a well-defined ownership model.
[0] ...implying the existence of distinct parts that can be independently developed and independently fail...
I think they were against over engineering because they were constantly firefighting bugs and didn’t have time to thoughtfully review.
It’s also tightly connected to a normalization of lack of accountability.
> This isn't "getting an FTP account, mounting it locally with curlftpfs, and then using SVN or CVS on the mounted filesystem" -- you still have to do the hard part.
This is probably losing the younger portion of the audience by now. ;)
If you are pitching that your service can do potentially "whatever the client wants" you have such a thin basis on which to provide contracts and guarantees as a provider. The narrower the function, the clearer you can be about what's supposed to happen and why things might have gone wrong.
When you're using probabilities as the fundamental approach to computation, all of that goes out the window. Nondeterminism is powerful because it's insanely flexible, but the cost of that flexibility is predictability and expectation. Determinism was humanity's primary choice for formalisms and technology precisely because it reduces complex problems and situations to repeatable mechanics that are easy to understand. Deterministic tools can't do a lot in the grand scheme of things, but it is precisely these limitations that make them work well in concert and keep them comprehensible.
I bought a new electric car recently. For the most part I've been quite happy with it. Shortly after I bought it, it started popping up a warning message saying "check EV system" every time I started it. By the time I brought it into the dealership, the warning had gone away, and the technician just told me something to the effect of "eh, I guess it just does that sometimes, let us know if it happens again." Hardware fault? Software bug? Who can say?
Like most modern cars, it has connectivity and Google Maps built into the infotainment system. The vast majority of the time, it works fine. Sometimes it says it has no connectivity (meaning no traffic data and suboptimal routes) for the duration of a drive, even in areas with a strong cell signal where it normally works fine. Sometimes the car says it has connectivity, but Google Maps still thinks it's offline. Sometimes Maps will actually load and display a route, but the "start navigation" button just spins forever as though it's still waiting for something. Are these related issues? Is there a common cause that might be fixable? Who can say?
(Conveniently enough, the warranty specifically does not cover any failures of software or firmware to operate correctly.)
These are some of the most annoying messages a piece of software can possibly display, and I hate them with a passion!
It's like this with a lot of commercial software, where designers don't want to scare users with complicated (useful) error messages, so you just get a "whoopsie! Something went wrong".
I'd much rather have an error message that I _don't_ understand over an error message I _can't_ understand.
In the case of cars, it used to be that a generic "Check engine" light showed up on your dash, and you needed an external tool to ask the computer what caused it to light up, but why would that be nessecary in a modern vehicle? You've got a large, high resolution display (or more!) on the CAN bus, why can't it show the full problem so you could take a picture right as it happens and send it to your dealer for instant diagnosis?
My stance was that far too many of our support tickets involve someone taking a screenshot, and we'd want that information in any such screenshot... Especially if the network was so bad that it couldn't reach us any other way!
Fault codes are uploaded to the manufacturer on modern cars with telematics. On mine, the app will say you have an engine fault (or need an oil change or recall) and offer to schedule an appropriate appointment. If you call them, they often have it already in their portal under your VIN. The manufacturer also keeps track of these for regulatory purposes (triggering recalls).
A single fault code doesn't mean much. That's like saying I got a segfault, please fix my problem. You need a stack trace, registers, what the user was doing at the time, whether they can reproduce it, etc.
OBD2 stores a _freeze frame_, most of its readings at the time of the fault. You also need to look at other fault codes, pending codes, frequency/history of the fault to make a diagnosis. In any case, you can't just give one code; it's 5 pages of readings.
But my point - this has been increasing for quite a while and it's something like the evil twin of the "it just works" school of design. Because when the default expectation is "just working", the failure mode is "it just doesn't work". I mean, the default assumption that users are morons has de facto made us all morons since we can only deal with the "interface made for morons" and makes failure into opaque failure.
It's world the developers know "all abstractions are leaky abstraction" but where developers still won't give users (or even downstream developers) tools to deal with the leaks 'cause a simplistic interface matters more.
"When there's no emergency exit, there can't be an emergency, can there?"
> The major difference between a thing that might go wrong and a thing that cannot possibly go wrong is that when a thing that cannot possibly go wrong goes wrong it usually turns out to be impossible to get at or repair.
I think part of the answer is product management, which tends to get lost when the conversation is only about tools and code. Good product work isn't about how the software gets built. It's about helping a user do what they're trying to do better, faster or more enjoyably. The author's confidence-threshold example shows this well. "0.9 sounds about right" isn't really a technical decision. It's a product decision that nobody made on purpose.
For me, the real takeaway is that we will probably find a new balance of crappy, cheap software that people are willing to accept. However, I think that good, talented companies will still choose to ship good, high-quality products because it matters to them. The will not value simply shipping more, they will still be focused on achieving their goals.
Wasting your time and resources is a power signifier, but getting you to waste your own time and your own resources is hegemony.
Not sure why anyone would feel the need to dunk on this thing without showing a real failure example.
So they’re not dunking on that, they’re dunking on the idea that you can just trust Jev confidence scores without actually doing any validation.
I agree with that Evals are a scarce commodity rn. A business needs to define what good looks like. This is a laborious, and sometimes politically controversial, process.
Given good Evals, frontier llms are a magical tool that can automate tasks and do them more accurately than human ops teams.
Exactly which llm to use depends on the mixture of speed, cost and quality of the output.
Jev makes a claim to expand some regions of the Pareto frontier. I look forward to testing if this is true.
There are many areas of work we can’t automate rn. We cannot create good Evals either because time horizons are too long, or it’s too difficult to create good Evals.
That doesn’t mean there’s anything wrong with building good ai engineering systems in areas where it works magically.
Without the evals, how do you know it works magically? I think that’s the entire point of the authors article, people are using AI tools, and trusting the AI own evaluation of its performance, without making any attempt to validate if the AIs confidence actually correlates to anything meaningful.
From what I’ve seen, the result tends to be very exciting demos, followed by incredibly mediocre results when deployed into a real environment. The lack of critical evaluation then provides a double whammy, because it takes longer than it should to realise the system isn’t actually working correctly, and then the team lacks the skills and understanding needed to improve the system they’ve built. You just kinda have to hope that the next model release is going to magically result in better performance for your particular scenario, which won’t be able to measure due to the lack of evals.
This can be determined by offline benchmarks if you build a system that takes small sequences of actions, and requires live ab test for long sequences of actions.
The more your human ops team work from documented standard operating procedure, rather than tacit knowledge, the less you need Evals except to capture edge cases
"Normal accidents, or system accidents, are... inevitable in extremely complex systems. Given the characteristic of the system involved, multiple failures that interact with each other will occur, despite efforts to avoid them... while operator error is a very common problem, many failures relate to organizations rather than technology, and major accidents almost always have very small beginnings. Such events appear trivial to begin with before unpredictably cascading through the system to create a large event with severe consequences." [1]
It should be impossible to die from "poverty" (certainly not in a "rich" country like those in Europe or North America) and yet that is one of the most significant contributors to a staggering number of ongoing deaths. Things get only worse if you consider stress from existential anxieties or conditions that would have been treatable if they had been diagnosed earlier (but that would have cost extra).
Well, I guess that's not much of a consolation after all.
Of course you aren't going to use Jev where you can easily make a decision deterministically with old-school programming. It's going to be used for stuff that might have previously been done by a person, e.g. reviewing app updates in app stores, reviewing CVs, labelling bugs, etc. Obviously that can already have inexplicable failures.
It’s driven by a large “I want to believe” factor. A model that can do some of what an LLM can do, but is smaller, cheaper, faster? There’s huge demand for that, and Jev has exposed that demand.
I don't mind doing it but why is this the norm
Literally go no further than the age old advice of "have you tried turning it off and then back on again?", and then that actually working.
Did this person never experience the effects of rocking the boat just a little too much? Daring to do a little too good of a job? How?
If so, it's definitely newish for those of us who started programming twenty+ years ago. Back then waterfall was the way to develop applications. Once it was burned to CDs or DVDs, that was it. We could not afford shipping broken things.
Ever since internet became ubiquitous we moved to on-the-fly patching for installed software and fast deployment cycles on cloud.
That said, the lack of forethought in design and backward compatibility is nowadays definitely on the rise and occurs likely on an order of magnitude more often.
Or the phenomenon where I piss blood explaining why having a p99 that is 30x worse than our p95 is maybe possibly a problem that should be looked at.
Or the phenomenon where concurrency exists, and so issues are no longer reliably reproducible, meaning everyone just throws their hands up and tries to ignore and downplay them as much as humanly possible. That is until a dickhead like me comes around, and does something like a scripted 300 restart cycle test overnight until a clustering resiliency defect finally reproduces, and i can capture enough debugging data that would never be possible on a live environment. All the while the product vendor is twiddling their thumbs, waiting for us to provide said data on a silver platter, because for some reason this completely stock issue doesn't reproduce on their end in a pretty much identical environment, or so they say.
I feel your pain and have had the same sort of issues. I once spent 3 months on a hard to reproduce issue to prove it was the closed source software from a vendor that was at fault.
Bugs have definitely been present since the start.
However, since we moved away from waterfall (not that I really miss it), systems are not that carefully designed anymore. Even if at the start the system is well designed, careless additions start creeping in and before you know it there are bottlenecks and race conditions everywhere. I think this was caused by the "move fast and break things" and the reduction of QA teams.
In other words, if it works in my small and non-representative environment, ship it. If it breaks, someone will tell us and we will fix it. I don't like this either, but yeah, it's been like this for a while now.
Latest development is IMO moving even more in this same direction, so it's getting worse rather than getting better :(
Lovely post, and lovely phrase (normalizing of inexplicable failures).
This is just the complete opposite in my experience. Tests are the first thing the AI writes, especially in low coverage or unknown domain situation.
These systems have been trained for "generations" to oneshot problems. The first thing they do is create a mini harness to verify their solution is correct.
Majority of issues with AI assisted development are misaligned requirements or poorly stated goals
Excellent post!
(Yes, storing gold on a what the room door claims is a toilet is a best practice, as thieves won't look for it there. But now the security through obscurity got leaked /s )