AI handles incidents, engineers lose touch with their systems | Hacker News Reader

AI handles incidents, engineers lose touch with their systems

sylvainkalache.com

418 pointssylvainkalache351 comments

A natural evolution of engineers losing touch with the customers and users.

I'm noticing some of the concern play out regarding AI weakening the capabilities of software people.

I gave the team an exact solution on a silver platter and they still failed to identify how to go about it after 3 days slamming it into Claude. The resolution is literally 1 line of code that could be arrived at in about 30 minutes of patient, old school troubleshooting.

I think what's happening is the AI system draws poorly aligned and led engineers into this ego inflation feedback loop where they are completely detached from reality because these tools can simulate a better one.

I'm seeing this happen in the security space right now. Someone on my team I was helping train and bring along is all of sudden regressing in their understanding of the issues we're working on, and instead focusing on AI tool outputs to do their job for them.

I can share a weird story:

Usually, I take my time to understand each keyword of the code I'm looking at, especially if it is new to me, like terraform.

I work in a team/with one architect, who only did the DevOps/Infra stuff for the past years and I had the expectation he knows what he is doing and talking about.

At around 2 weeks, I noticed how his knowledge has severe gaps and how he takes things at face value or uses terminology interchangably, which confuses me. It sounds plausible, but it does not actually translate into a working system or shared understanding.

Then one day I did some pair programming with him and whenever there was an error or a resource missing, he would type it into the LLM, copy paste it out of it and then brute force error messages. He did not even wait a second to think or reconcile whats happening on the screen or what the exact requirement is. Never taking one step back and questioning any assumption.

Now that the timeline is shifting and everyone starts to be stressed, he continues to vibe code through me and it is so tiring, there is no higher level planning or architecture, its just a reactive type of trial and error to be faster. It feels like these people are so used to talking to bots, that they treat you like an agent they can chat to or talk through monologs with.

It is quite shocking how people went from being humble (learn the basics or close the gaps in understanding) to full on authority on everything and berating people 24/7...

So right now I'm considering quitting IT for a couple of years until people calm down, but I think its pretty futile

You captured this phenomenon very well in this comment. Appreciate you sharing it because it’s hard to describe exactly what makes this sort of behavior so bizarre.

@backlava12 fyi your comments are all dead in this thread

Great comment.

At my place, this is what they want. They want people to smash through things as fast as possible. They don’t want people to sit and craft a solution which takes in to account the whole. They are choosing tools which are low code, and use llm’s to produce what they need. as they say “this is the way things are going”.

Even that would be acceptable. But they still blame the programmer, who was pressured into this, for any fallouts because of it. Basically trying to have it both ways. The unreasonableness of it is infuriating.

Pretty much this, they want to impose one method, but they don’t want to own the mistakes that it creates. Sometimes it felt like the bullies that are saying “why are you making me hurting you?”.

I see it as laying the groundwork for firing. This is bound to blow-up in production eventually which would be a good enough excuse. If it doesn't, reduce token budget and squeeze some more until it breaks.

In today's episode:

"Programmers frustrated that they're being treated like craftspeople or delivery drivers or factory workers, etc, etc."

I don’t understand how this comment is productive for any of the groups involved. Can you explain it to me?

It doesn't try to be productive.

It's exposing the hypocrisy of privileged groups suddenly being treated like regular people, when they did NOTHING for decades to help others or even to protect themselves ("f** you, got mine", rockstar programmer, 10x programmer, talents everywhere, etc mentality). No unions, no collective bargaining, sheer individual greed.

Do you think that all programmers have always existed in the US?

On that note, do you think that all programmers in the US are these Silicon Valley stereotypes?

Plenty of programmers in the US make normal wages for professionals with degrees doing things not at the technology companies famous for their disregard of the rest of humanity.

As someone who has done trades and also “machine shop programming” and jail breaking rural equipment for friends to use I’m wondering why comments like this are focusing on dividing us instead of bringing us together.

It's a universal programmer trait. Let's call them "culturally regular programmers": they are less social than the average person. They lean more libertarian and more selfish.

According to who and what?

I think the entire open source community would like to have to have a word with you.

The Open Source community is programmers sharing with other programmers, and it's largely fueled by corporate interests these days.

The Free Software movement has largely failed by now, as shown by GPL 3/AGPL adoption.

And again, all of this was based on the fact that programmers were a "special breed" of people coddled by corporations. Now that the corporate mask is off and programmers are no longer (as) special, the whining is horrendous. And the libertarianism still won't go away.

> According to who and what?

To me, I've been around the block for several decades.

Again, not everyone, but the culturally regular programmer.

It's a snarky way of suggesting that programmers are complaining about being treated the way professionals and tradespeople always have.

Yeah I get that. But having been both it doesn’t advocate for better treatment for any of them.

Also as far as I’m aware the pay for programmers in the US has always been an isolated thing with the rest of the world ranging from normal to “literal sweatshop”.

What do you mean, "advocating for better treatment"? They're not describing an abuse of tradespeople and professionals. They're adults doing a job.

So, from your comment and your reply, are you just happy that more people are being treated poorly?

That makes no sense. You seem to understand that he was being snarky, so why do you not understand that that snark and desire to see “others” brought down to a lower level just because some were already there doesn’t help anyone? Actually, that division does help people, those people that are putting people there in the first place.

Let’s be clear: for context I’ve been out of the programming game for more than 10 years at this point. I have my own business that I’m set to sell soon and then retire to something more entrepreneurial. But I still don’t desire to see my younger employees suffer any more than the tradesmen I used to work with.

You haven't established anybody being treated poorly, only that they're being treated like professionals and not artisans.

Haha, it seems childish but if you’d like to go that route I’ll just point you to the original posters reply:

>It's exposing the hypocrisy of privileged groups suddenly being treated like regular people, when they did NOTHING for decades to help others or even to protect themselves ("f* you, got mine", rockstar programmer, 10x programmer, talents everywhere, etc mentality). No unions, no collective bargaining, sheer individual greed.

So, I guess since you’re taking that position, Thomas: are you just pro pain for an increasing number of workers or can we agree that adding more people to the pool of mistreated labor isn’t beneficial for anyone?

Personally I’m going to continue advocating for better rights for both.

What "pain" are you talking about? You keep trying to smuggle this framing in. I don't accept the premise.

No smuggling needed. The original poster and I are on the same page and I suspect you’re just playing games to avoid hard questions.

I’m assuming from that that fly.io may not have a great work culture.

What a weirdly hostile and offensive response. I'm always struck by people who think they can namedrop my employers to shut down an argument, like we're just going to believe and profess whatever keeps people on message boards from being mean about the company.

But, no, I feel like throughout this thread I've been pretty consistently putting a question to you (what exactly do you mean by "pain" or mistreatment, given the original comment described other professionals having norms developers seem now to be recoiling from).

We can be done, though, obviously.

Nah it’s relevant, because if you hold these views to the point that you’re willing to avoid answering the question then obviously your world view is warped in what even you would consider a negative direction and therefor it’s important that others get a chance to be steered away from a potentially negative experience.

And to be fair, I don’t know what you expect when you advertise your place of work and then insensitively play games with serious subject matter on public message boards. I think the kids call that a self own.

But I agree, you don’t seem willing to put in writing what you’ve conveyed so this isn’t a growth experience for anyone.

reading through this it is obvious what the "pain" is and it does come across as you just avoiding the question. Just sayin

Maybe you can answer the question then. Is it "having performance judged by output rather than by process or satisfaction"?

I don't think it's generalizable. The kind of person who copy and pastes from the AI is the kind who did the same from StackOverflow before. It's more compelling, and we probably see more of them because of it, but it's the same general thing.

The kind of person who insists on understanding things and working through the problem has always been rarer. It's not "humble", it's "inquisitive" and "persistent".

I'm seeing people who _used to be_ like that losing that understanding without realizing it's happening - they have a superficial idea of what the code is doing, enough to feel like they understand it, but the change is apparent when watching them handle something unexpected.

LLMs reduced interest on tech debt, but it's still there and people who have a tendency to acquire it will go bankrupt eventually.

"The code market can stay irrational longer than you can stay solvent." will probably be a thing.

That's certainly what I expect to happen. LLMs will ruin your software if you don't hand-hold them, but it'll take time for that to happen. And until that does happen, businesses hell bent on using them won't have anything to push them into realizing what a mistake it was to use them.

Two imagined executive responses:

1. "Nonsense, if it was a code-quality issue from the AI stuff, we'd have seen problems sooner, like in the first few months or a year. Oh, sure, some engineers complained, but that was just an adjustment period because they stopped once we told them we didn't care and mandated that they up-skill into the new AI-centric workflow future."

2. "The bold new AI initiative I put into place can't possibly be wrong, this failure is on the engineers who were responsible for overseeing its operation. This is supremely disappointing because we made their jobs so easy, they no longer have to write code, just review several dozens of pages per day with unremitting paranoid vigilance and attention to detail."

This, plus anyone responsible dodging that as soon as possible via job hopping. Harder to do in the current job market but internal transfers are still available, most of the time.

I was pretty early in the LLM stuff and I got called out on using it back in the day, because I showed the exact symptoms we see now, but as soon as management jumped on the train all the coworkers did a 180 and I alt tabbed out into a different department thinking it would change something... well it didn't :'D

I guess it will take some years till we are sane again or not...

I revisit it to see if the promises are unbroken now from time to time with every other update and these days it doesn't look so good. It is non deterministic all the way down.

The difference between the two is that friction is learning.

the less friction the less growth.

So even if they copied from SO they would have better knowledge than an Ai user.

This isn’t speculation either, theres an MIT study which this is based on. https://www.media.mit.edu/publications/your-brain-on-chatgpt...

Learning could be targeted though.

If I know where the error is (a script written by AI in powershell) and it's a logical one, it's ok to let the AI reason about it to get it fixed and move on.

If I have no clue about how this part of the system works, then it's worth reading closely what the error says, in order to understand it first, then have the agent check the assumption you have.

Reduce the friction by having the agent explain what happened and why the fix solved it. I know this might be an intellectual placebo, but sometimes you need to fix something fast to move on. Learning takes longer, and these days everyone expects you to be a 100x engineer with AI.

In essence, you are correct. What I see with LLMs is how they will happily make up stuff, meanwhile you can pretty much exhaust SO, until you are forced to understand the problem you are having.

You don't have to be a certain person to fall into this trap, you just condition your brain to accept this workflow somewhere.

Pair this with day to day work stress like time and amount of tasks and you almost give in to a sort of addiction to deal with it all.

> I don't think it's generalizable. The kind of person who copy and pastes from the AI is the kind who did the same from StackOverflow before. It's more compelling, and we probably see more of them because of it, but it's the same general thing.

There's an infinity between someone who has to panel beat what they copied off SO and someone who just bangs their head repeatedly against an LLM.

> The kind of person who copy and pastes from the AI is the kind who did the same from StackOverflow before. The kind of person who insists on understanding things and working through the problem has always been rarer.

Well, this is where I disagree. I have coworkers who used to insist on understanding and now are doing exactly as OP wrote - copy/pasting from the LLM to brute force error messages. I don't know if it's generalizable but this is what I see in <big tech> working on a frontend team with mid to senior level engineers who I respect.

One example a few weeks ago, I was helping a coworker root cause a bug in a React codebase. Pair programming isn't necessarily common but sometimes you see someone banging their head at a problem and you get curious. It turned out there was an effect (a callback that's invoked whenever some state changes) that invokes an API and this effect caused an infinite render loop because the error handling wasn't written correctly. It was something silly like - Call this API if we have no data -> get error -> update state -> call API again because we have no data...

That was almost immediately what I suspected but my colleague was pulling out all the stops usually reserved for when you're desperate or need a sanity check, like logging to stdout after each line. Both my colleague and the LLM were convinced the problem lied in the pagination logic of the helper that invokes the API. He ended up rewriting that helper imperatively and functionally. To his credit, he rewrote it by hand and implemented the recursion correctly but he was baffled when the problem remained. Completely surprised Pikachu face.

I don't think my coworker changed or suddenly stopped caring. It seems much more likely this is a predictable outcome when you lean heavily into AI authoring code for a sustained period of time. I'd also say that in itself is a consequence of the extreme pressure being exerted across the entire company to ship more code and review more code, faster.

>> he continues to vibe code through me

So quit pair programming. I never did, never will do that, nor worked at a place that remotely encouraged that. Each to their own, that's how it should be.

I wish this was always possible, but sometimes you do it when stuff is ambigous or the task is almost undefined and you just can't reach far enough for someone to fix it, so going over it and talking is simpler

> It is quite shocking how people went from being humble (learn the basics or close the gaps in understanding) to full on authority on everything and berating people 24/7...

That was never humility. That was fear. This technology is just enabling people to show their through colors. The unearned confidence is a hell of a drug.

I don't think this is going to change because it helps automate code production. As in, once we reach a point where this method produces acceptable results, we can fire all engineers and automate the process.

At least at my company the OKRs are quite clear and demand heavy AI utilization above all else

In my mind, this, not copyright or water use, is the best reason to boycott AI. It'll make you incompetent.

I saw this the past year - new employees would put problems into Claude first instead of debugging. A year back I was debug manually first, now I do the same.

The speed AI debugs at is incredible and yes, we lose touch the more we use it like any manager feet up barking orders to their underlings to get things done.

To be fair the cyclical nature of funding and defunding security teams which causes scaling up and scaling down, has always left a race to the bottom in security.

Any CISO or head of security loves the ideal of completely using AI to handle incidents, tune detections, implement mitigations, track vulnerabilities, pen testing, etc. This feels like it driving security teams to have less critical thinking.

I it usually doesn't get me in this weird state of mind, but I once spent 6 months (all-in) building a thing that I, once finished, just left alone completely (on disk gathering dust). Weird experience. So I'd say AI physchosis is real.

I think LLMs have some of the same risks and benefits of stimulant drugs. They can make you more productive if used effectively as a tool, but they can also delude you into thinking you are better than you are and create a dependence such that you aren't just less productive without the LLM/drug, you fail to be productive at all because you don't know how to function without it.

That sounds somewhat applicable to many tools. Like Vim/Emacs, for example. Or computers and smart phones in general.

Yeah, like caffeine is to meth

Thank you for this, the hand waving comparisons of AI and anything else is getting out of hand on HN. Your ”like caffeine is to meth” is one of the more succinct and describing sentences I’ve seen describe the difference.

Computers - mostly no.

Smartphones - mostly yes.

Smartphones are fully portable which makes them always available and super addictive.

Computers have 10% of that addictive power.

If the solution is so simple, why claude did not found it? At this point we can assume, it is better than 90% of engineers (including me).

After three decades of outsourcing to lowest bidder, I do not buy that humans are somehow better!

> patient, old school troubleshooting

I usually see similar arguments around systems with major red flags (no docs, poor CI, decade ago no CVS...). And engineers with private stash of workarounds for job security!

Claude does not do anything special.

Or perhaps claude was misconfigured, it had no access to relevant part of system, and it tryied to work within its limitation. Often it means decompiling binaries in desperate loop...

Claude regular spits out six helper functions instead of... A twenty line for loop. It overengineers most things.

Overabstracting, deduplicating things that don't need to be. Building metaclasses because it saw a single orchestrator in the whole codebase.

If it is a better engineer than you... You need practice.

The funny thing is that if you never understand the codebase then you will keep thinking Claude is doing a great work delivering all this incredible software, when all it has done is created unnecessary tech debt.

Which at some point the developers who actually still do know how to program are either going to have to clean up or in terminal cases rewrite.

In a way all we've done (currently) is drastically expand the amount of technical debt across the whole industry.

Should be profitable for the ones who can still actually program though and haven't let their skills atrophy by letting Claude do everything.

I don't deny there are use cases for LLM's, I just don't buy the hype about them either.

As with all tools, you have to understand how to use them to get done what you need to get done without sticking the chisel through your hand.

But this is leaving out the part where the developers that clean up or rewrite... will do it using LLMs.

Have you tried refactoring or porting codebases larger than a million lines of code pre-gen-AI and again post-gen-AI? It's night and day difference. One would be insane to schedule a team on 8 months worth of grunt work porting from one language or framework to another which can now be done by 1 person in 4 weeks.

Of course the person driving it has to tell it exactly what to do and has to have the requisite knowledge to understand how to effectively structure or fix the software. Maybe new developers don't build this skill so easily anymore. But I don't see why a strong developers skills would atrophy in this case though unless they just never use their knowledge and never give instructions to the AI.

To developers speaking of skill atrophy: are you still making sure that when using LLMs you are actively exercising skills like system design, debugging, reviewing for clean code and just in general doing effective code review? If you are doing that, why do you feel skill atrophy? And if you aren't doing it, why not? What about LLMs prevents us from exercising these skills?

2040.

The demand for real programming skills will become infinite (again).

someone who can actually read, understand, and debug code. when the clankers get stuck.

unfortunately, there will be only be a dozen people.

I am former java enterprise dev, so yes I often code this way. Unit testing, decomposition... Some projects CI refuse to merge commits with 20 line loop and duplicated code...

But that is not a point. Claude can code tight compact loops, it just needs to be instructed to do so! If it does "enterprise code", it means it had no instructions about code style.

If your documentation, spec, agent.md does not have proper guidance on coding style... yet another red flag!

> it just needs to be instructed to do so!

Considering how often it overrules, its own rules?

> A twenty line for loop. It overengineers most things.

Anecdote I like to tell.. I was working on a financial planning software, intentionally purely vibe coded as an experiment.

I eventually discovered AI had implemented seven duplicate copies of tax calculation functions. All of them different. All of them wrong. All of them giving different answers for same input.

Not even the most junior of newbie junior engineers would do something this crazy. But AI was happy to do it. It will solve the immediate problem, efficiently. Even if the most efficient solution is something ridiculous like this.

For some definitions of efficiency.

I have also a weird story to tell that a human did and it is as crazy as this. It happen in 2019 so no LLMs at all.

A person that was hired as an expert in our startup spent more than one week full time working on implementing his solution to the problem we were having. I checked the code after one week to see the progress and was curious how they are implementing an already crazy sounding idea. I found that the whole week was spent re-implementing in python, python's built-in "float" function. That was it, the whole code was just that.

Our problem was related to financial services and their implementation of "float" was not even correct.

Jesus Christ... and then?

What happened to him? What was the conclusion? Did he get paid?

of course he got paid.

that is the deal for an employee, you put in the hours, you get a check.

today, this individual is CTO- Vibecoding at Uber.

ah. when you wrote „hired as an expert“, I misread it as him being a contractor.

nevermind

It's funny you mention this, I've worked for financial type systems where they spend considerable time removing any floats the from the code base.

I'll leave it up to the reader to figure out why this may be important.

Which model though? I have similar anecdotes but all with older models. The jump in capabilities in the last 6 months has been substantial.

I have literally been hearing that, over and over and over, since '22.

And whilst it is obvious things are growing... Saying that, sounds almost entirely like the person saying it cannot objectively look at the environment. If everything has changed in the last six months, why has the industry not radically changed to match it?

Everything really did change with the Pentium II. It did with 3dfx. It did with Damerau's taken on Levenshtein. Hell, everything changed with React. The AI leap with seq2seq completely revolutionised the entire industry. But... Its kid, the LLM? Really?

> The jump in capabilities in the last 6 months has been substantial.

What I would like to see is a chart graphing the model size against some objective measure of capabilities, specifically for coding.

It's easy to see gains when you're doubling the effort. What I want to know is if the extra effort is opening up more capabilities over time or fewer capabilities over time.

They would not do it in the span of a day or a week. But I’ve definitely seen something like that happen over a period of multiple months.

The llm just allows to generate faster.

We can feel smug about that but all it means is that we need to be clearer on our requirements and preferences up front.

State that similar functions should be in one place and there should be only one. Today there has to be compelling reason why that function is different from others. Normalise the function name based on what it does. Why are there different ones?

Then there are all the other guard rails in place.

Better guidance from mentors, reviewers, and automated project tooling helps everyone. Juniors, seniors, and engineers.

Oh and how is it any different than most software engineers?

How many times I heard ORM are bad only to recreate the same shit?

How many times I heard ORM had bad performance and see 1+n stuff everywhere?

How many times I have seen tight coupling in the name of DRY?

Its different, in that when you teach that engineer, they either leave because now they hate you, or they grow. They change to meet the standards of a project, rather than inventing their own.

We don't get seniors, without juniors. I'd say more than half the job, is just... Learning. People grow.

Exactly. People ask how we get seniors with juniors using llm. The answer is the same. Review the code. Analyse write down what is wrong. What you expect to have been better. Force every change to be documented and explained enough.

It just takes forever now. The understanding is lower, the effort is lower, and frankly, I think the interest is lower too. I might be in the last generation who truly had fun working on a 'shrodinger' bug.

Yes. So find someone with intrinsic desire to engineer. Mentor them.

In the mean time put a plethora of guardrails in place to make sure the AI Train doesn’t derail production.

Oh. And keep showing your value. In the end every org can do with less low paid overeager uninterested juniors. Might as well let agents Do those tasks.

> Its different, in that when you teach that engineer, they either leave because now they hate you, or they grow.

Or they just have their own hubris and ignore your (provably better) suggestions because their way is "better/easier/how we've always done things".

And then you end up with someone sprinkling N+1 issues throughout the system and making systems with bad architectures throughout the years, not thinking about backpressure etc., as well as shoving ALL the dependencies into a single codebase cause they're not used to creating new ones, turning patches into eventual month long version upgrades because everything keeps breaking with anything newer than JDK 8 and some of the packages are deprecated and gahhhh I should pick up woodworking as a hobby.

Though, to address the original claim:

>> If it is a better engineer than you... You need practice.

This feels like a thought terminating cliche. Like, it will spit out bullshit every now and then, and make assumptions that I don't think that many engineers would (e.g. since a lot of each app is environment-specific), but at the same time when you guide it and give it examples, it can really be quite good! So not that unlike humans at all, even competent devs might not necessarily know about every pattern in any given codebase, especially when one has been around for 10 years and grown quite a bit.

It can be quite good if you have something like ArchUnit or your own tools for linting project architecture and patterns, alongside proper documentation that doesn't assume that you're a team member with X years of experience on system Y. AI just forces people to be less lazy and ignorant about knowledge transfer, which they should have also been for the sake of other humans!

I have never had it be every now and then. It is always bullshit, the first time around. Usually followed by, "You ignored the first three rules, and all the examples, try again."

I have worked with some amazingly incompetent devs. Some promoted into place to become someone else's problem, and some parachuted in through connections, and never once have they brought the continuous and unevolving frustration of modern LLMs.

I taught one of our "React SME"s, what Typescript even was. And they, were less likely to throw a ten thousand line fix, where twenty would do. They didn't see that the DB had a list of validation rules, and copy and paste those into seven different files, instead of just querying the DB, so the code would be kept up to date. And whilst I might have had to repeat that the DD is our source of truth, they never argued that implementing the DD wasn't aligned to the intended design.

AI isn't forcing people to be better with knowledge transfer. It's taking conversations that happened in person, that were back and forth and gave both people a better understanding of a thing, and turning it into a half-assed and out-of-date wiki. Without all the things that make a wiki actually useful to onboard the next newcomer.

I do get what some of these frustrations are, and where they come from. I'm the official documentation maintainer, because nobody else ever wrote anything down on expectations. Its fairly thankless. But... All I've seen AI do in that same realm, is exacerbate misunderstanding.

Like when it presented an Apex script (Salesforce) guaranteed to exceed governor limits to our junior, who took it and ran with it. And when it aborted, the AI spent half a day leading the junior around and telling them to change config settings - that the environment was the problem, not the code.

The fix? Toss the three layers of Queueables, and use Database.insert on a list. Job done. PR for 150LoC, instead of the offered 780LoC.

(As a complete aside, can anyone convince Salesforce to fix the documentation for: https://developer.salesforce.com/docs/atlas.en-us.apexcode.m...)

Thank you for the Archunit recommendation, that library looks incredibly useful in this modern, agent driven world.

100%

It is my pet peeve with Claude and why I don't prefer it for most stuff

(also the comment spam - but that's a all of them in a way or another)

That final 10% is the hard part. 90% is easy.

I think people on this website have a way, way to high opinion of themselves. Barring a few exceptions, we are middling developers, or way worse than this. Just read the comments from a year or twoon how chatgpt4.5 is the best at coding or how Claude sonnet 4 is better than a lot of Devs... At least I don't have imposter syndrome anymore.

So after 30 years of outsourcing to the bottom 10%, you think Claude is better than the bottom 90% even though it’s so stupid that it doesn’t even know it should ask for advice or more information when it’s stuck?

It just follows instructions you give it.

Some asian devs will go for weeks without asking for help, all while giving amazing fake status reports. Loosing face etc...

You're comparing scammers to incompetence

Do you think those devs are in the top 10% of all devs like you said Claude is? Or is the bar suddenly much lower after all?

Claude asks me all the time when it's stuck, what are your prompts?

I think some of these anti-claude comments have some face, then I get a peek into how people use them, and it seems all wrong.

The tooling is incredibly configurable, but comes out of the box with a big auto button, because thats what people want. Doesn't mean advanced users have to use it.

> At this point we can assume, it is better than 90% of engineers (including me).

Hard disagree. We absolutely cannot assume that. You can posit it, and we can have an informed debate about it. This is what irks me the most about LLM fans: they constantly try to reframe the debate to have their worldview as the agreed-upon starting point.

While I actually agree with you (though, outsourcing to lowest bidder would account for much of what you're seeing with humans), I just saw Bug Hunt Bench scores that gave me some pause:

https://x.com/PawelHuryn/status/2095982259761475945

https://bughunt.productcompass.pm/?preset=all

Claude Opus 4.8 ranks near last on this Bug Hunt benchmark, and missed 96% of the deliberately introduced bugs. If you're a developer who has been falling back to Opus 4.8 because of how Opus 5 talks, and Fable 5 being so expensive that it needs to be rationed... well, turns out Opus 4.8 can actually be quite poor for finding bugs.

(Which feels weird to me, because Opus 4.6 fixed a bug that myself and a group of humans had been hunting down for over a decade. Models are spiky.)

Also surprising to me: Luna Max performing better than Fable 5.1 High, at least on this benchmark. But Astra 6 & Fable 5.1 on Max both perform at the top as you would expect.

Still, basic debuging and trouble shooting is where LLM generally shine. Any model can bisect git history and isolate newly introduced bug.

If model can not automatically reproduce bug, while human manually can... you got a problem in CI.

> Luna Max performing better than Fable 5.1 High

Perhaps you are reading too many benchmarks.

Edit for answer :

I agree Luna is great cheap model.

But if Fable was hitting security limits, yet was still included in benchmarks...

What flies better? Elephant or paper plane. You can make objective benchmark about that. But not much value for logistics company

> Perhaps you are reading too many benchmarks.

Maybe, but at least the benchmark provides an objective measurement of the codebase it is tested on. You're also assuming the bugs are newly introduced / regressions.

I can give a concrete example - Fable will not interact with bugs that result in writing to null pointers in C code. That triggers the guardrails and ends the session. If Luna (or GLM Flash, etc) will fix those kinds of memory bugs, that immediately puts it ahead of Fable in some ways, no matter how tiny Luna is. Again, models are spiky.

I still agree with your initial point! It's LLMs all the way down over here. It would need to be a particularly gnarly bug & an exceptionally talented human for me to want to pay another human to work on fixing it now.

Because simplicity is hard and often the result of careful thought. Anybody can keep piling pile of shit on top of pile of shit which is why that sort of code is so common in our industry.

I don't think it's engineers, it's the rest of the org insulating the tech workers from every side of the business

I think there are many cases where it was the tech workers themselves who argued for isolation from the customer so that they may focus harder on whatever tasks. I used to be one of these workers. I argued very hard for it. I regret that today.

On the surface it seems rational, but it quickly turns into a system of perverse incentives because now the development team must maintain an illusion that they are constantly overwhelmed with tasks and could never hope to spare a microsecond to assist the customer. This misalignment is how you wind up building your own web frameworks and databases from scratch. It turns into a self serving monster that eventually dominates the entire business. From the perspective of the business, many of these development teams look like they're behind some modern day iron curtain.

It's the uncanny valley of AI. It's still not quite good enough yet that you can trust it blindly on a big codebase, so you still have to read and understand everything - which is often harder than just writing it up yourself.

EDIT: don’t get me wrong. I still think AI is incredibly useful for a lot of tasks! But when implementing an architecturally hairy thing, I find it less stressful and equally quick to jump down to the editor level and use AI just for code completion.

Pretty much, The one thing I use it for is as a sanity check, pretty much "Look at <SomeFile>, point out issues you see, summarise them tersely" and it'll spot stuff a code review by a human might have spotted (in the mythical land where people actually do code reviews properly and don't just flag a spelling mistake to "show they looked at it").

Beyond that I don't trust it at all and I still write all my code the meat sack way.

Trust is earned not given and it hasn't earned it yet.

If anything, I think this hype cycle is fast exposing just how many people, teams, and companies just don't care about what is correct. They just wanna feel good about themselves and get paid.

I for one welcome the fact that this whole thing has driven me back to books and deeper into the fundamentals. I have never read so much on math, hardware, and history as in the past 3 years or so.

Are you doing those things for your own enjoyment though, or to eventually capitalize on it?

And if it’s just for enjoyment, then doesn’t it make sense other people who want the same would just get a job where they can keep pushing things to an AI, feel good, get paid, then quickly get back to the hobbies they really love?

You are advocating for the kind of compartmentalization that got us here in the first place.

Nothing wrong with compartmentalization. Some people like it, some people don’t. Just different ways to live your life.

I don't think "compartmentalization" does this topic justice. Note, I'm not accusing you, since I don't know your work. My post is about the general milieu and the rhetoric around it. This "compartment" terminology seems to whitewash a dimension that has diligence on one end, and fraud on the other.

I've always worked in high-trust organizations where we depended on each other operating on the diligent side of things, making conservative choices to never wander into the murky area in between. It is horrifying to me how many people seem complacent about or even complicit in a different objective, which seems hell bent on wandering as far into the murk as one can without being caught.

When the person who has a duty of diligence starts rubber-stamping AI outputs, they're veering off into that same murk. They accidentally or wantonly trust the agent as if they have delegated their duty of care. But the AI tool has no such duty and no capacity to care.

I think this worker who has turned themselves into an outsourcing middleman needs to treat the results just like "found code" in a USB drive they found in the parking lot. Its origins and purpose are unclear. It could be flawed or obscurely inappropriate for the intended application, it could have legal entanglements, or it could even be subtly adversarial. The review task to figure this out is not simple. It is not something you do by skimming the result, or worse, asking some other AI tool to review and summarize.

The person importing such code to a project needs a different kind of diligence to try to screen it. For a lot of people, I think this review may be impossible or at least no less laborious than doing the original work themselves with the required diligence.

And, I think this importer needs to be fully liable and responsible for the outcome. But, instead, I think we're seeing frequent appeals to blame the machine and act like it is an honest mistake to let things pass because they've been rubber-stamping the imports. A lazy desire to claim credit for appearance of success, but shirk responsibility for detected failures.

This view is a little idealistic.

Software Engineers have historically not been very liable even for errors they create by their own hands, and companies tend to operate on a “blameless” engineering culture where there is not really a formal consequence for screwing something up. Your peers may not trust you for a bit, but the trust builds back rapidly as they make their own mistakes and then empathize with your past transgressions.

And in the AI era, this attitude has extended even toward code engineers aren’t writing themselves. Everyone agrees not to protest too loudly if someone commits something wrong because tomorrow it might be you who has done so. The days of “obviously bad” slop code are largely behind us and most recent LLMs tend to put out decent code, so it is very hard to catch problems in review phases, you’d have to be very eagle eyed and that’s not scalable to every employee. It is much easier to let the code run wild for a bit and just be ready to fight fires should they come up.

Much the same here - I have gone back to basics and am studying a lot more than I ever did.

My advice is to try letting the agent fill in the gaps. You can probably architect better than it can. Write your class outlines, explicitly define the public facing bits and what you want APIs to look like. Write the key integration tests that you know ought to pass.

The real advantage is that agents routinely write code without any silly copy/paste mistakes like accidentally accessing x twice on a coordinate operation instead of x and y. You can add some comments for what the function should do, throw in some real/pseudo code and let the LLM figure it out.

Your assumption is that LLMs will ever leave this uncanny valley.

Maybe unforeseen breakthroughs and different architectures are achieved. Given LLM fundamental shortcomings grounded in mathematics and information theory, I highly doubt they will and we will always need to deal with these issues in some capacity.

I think that bigger context windows help, but I feel that for AI to cross this chasm, it needs to be able to encode more abstract context knowledge. I think the holy grail here is online learning.

Agreed. For AI to be something you can trust to operate autonomously, it needs to actually be able to understand the things it is working with and reason about them. LLMs cannot, by their very nature, do that. There can be no reliability with such a tool.

I'd say it's more about learning how to organize your work more efficiently.

If you think about a product like marble: it's something that most be chiseled out of time.

Some people can chisel better products: the AI is just a better chisel.

Sometime still has to guide the chisel and judge the art/product.

In our cases, the market judges products.

I think my point is that its sometimes (!) easier to use the manual chisel rather than go for the automatic chisel and then fix its mistakes.

They say writing engages more of the brain and helps us to remember what's written more than if we just read it, or copy and paste.

When you say it's easier to go manual, it seems you're talking about learning retention. And you're right.

But seniors have learned enough that they're able to iterate quickly with AI.

They know how to organize their work, manage change, tasks. They know how to break a problem down into smaller pieces. They're aware of context windows, token cost, estimated task lengths, etc. And most importantly, and to your point about ease: they have less to learn so retention isn't an issue.

I have no opinion about whether we're in a good or bad situation, just making arguments from the toilet really.

It's not just learning is it, why do people buy hand ground coffee when there quite literally isn't any difference? Or audiophile snake oil? As long as humans are still the consumers, some part of consumption will be emotional. Could be to support local artisans, could be gullibility, could be love, whatever.

Maybe one day artisanal code will be a thing lol. Hand written like calligraphy. Those with refined tastes will have their favorite code artisans. And the plebs can continue with mass produced industrial junk.

It's already a thing among nerds. It's why Zig and Codeberg have their no-AI policies.

This seems to misunderstand the parent comment. The sibling comment plays along with the analogy and mentions having to fix the mistakes of an automatic chisel, but the original comment is alluding to the fact that it is definitely not an automatic chisel. There is no uncanny valley in an automatic chisel.

Programming up until this point was done by using deterministic tools to build products. LLMs appear to be nondeterministic tools in their current incarnation, at least to humans.

If an automatic chisel had a feature that could switch from chiseling from sedimentary rock to chiseling marble but would randomly and nondeterministically switch to the other mode during its use it would be considered defective. But with LLMs the industry has collectively decided that the nondeterministic automatic chisel builds so fast that the current defect rate from the nondeterminism is acceptable.

It would make sense to isolate the last line of defense from LLMs, ie the tests, but this rarely seems to happen any more. Once the tests are contaminated with LLMs all bets are off.

And people forget that along with atrophying skills and reasoning due to less coding, the skill degradation is hastened because the programmer is faced with the reality that they would have to constantly figure out, review or edit someone else's code (ie, the LLM's) if they truly wanted to maintain a last line of defense. But as this type of task is literally the least liked task in programming, the programmer passes it off to the LLM as well to avoid burnout...

I had a huge post written up to address your points, but I have to get moving, my ass is literally falling asleep.

It's easier just to say: It's all about time, my man. Nobody wants to waste it.

This. Time is money, and if you're not using an LLM you're wasting your employer's.

It's the harsh truth.

And of course, Google releases another dynamite piece of information to the world, with 'recursive dreaming and feedback loops'.

This revolves around an idea that started AI, 'Cybernetics'. Feedback loops that steer and guide based on input and output. The key to autonomy.

Google's whitepaper: 'Attention Is All You Need' was another dynamite piece of info that ignited and revealed something important about Intelligence: Attention is required.

And now finally, the last piece seems to be simply 'dreaming and self-reflection', observation about self. https://www.reddit.com/r/singularity/comments/1whwy4m/google...

This should push the edge along, making AI even more useful to us. Attention, plus a loop to observe, reflect, is the key to autonomous agency, whose system is governed by physical processes: speed of churning information and transmitting it to where it needs to be in order to make real whatever is it you need.

So really, the key is Attention (for Understanding), Time (to do Work), and Observation (to Improve). With these three things, the outer loop can steer itself (the aware 'thing', agency).

At that point, things get weird because there is no 'outer loop' anymore. The 'thing' becomes alive and is its own outer loop: a key idea of Cybernetics. The only thing affecting performance would be the speed of processing information and transmitting it where it needs to be.

And then: ensuring adequate quality of machinery, robotics, with minimal 'environmental issues', to help reduce churn (another feedback loop that would correct a robot's errors).

The outer loops decides, observes action, improves, re-observes, improves again. Rinse and repeat. What becomes important is Persistence.

For us, it's basic survival and emotional needs to some certain level of contentment. For a Machine, it's likely simply to be: to be useful.

Completely dependent upon us. The issue is Time, and how to fairly compensate everyone.

  > But as this type of task is literally the least liked task in programming, the programmer passes it off to the LLM as well to avoid burnout...

this is my daily life now and its miserable

its an overload spiral that gets worse at each iteration (even while 'gardening' our agents/skills)

the worst part is using the llm to generate the specs/features; its shit from the start and tumbles downhill from there...

> In our cases, the market judges products.

It happens, but it's rare. When last did a product take over a market without 100s of millions, sometimes even billions, of VC dollars?

There is no motivation to build a better mousetrap today, because the drooling idiot with a Claude account will look at how quick you signup users, clone it in a week (hey, it only needs to be superficially the same), and get VC money to dump until you go out of business.

I think the speed/context size of the large models is a threshold. I've been using a local model and watching it do killer stuff, and also shit out useless things; all in real time, requiring active steering.

I agree, except for the use of the word "yet" .

I think what's missing is fundamental. I think the reason it sucks so much to work with LLM-generated code is that LLMs will never "know" what it's like to be human. They don't "understand" our frustrations and motivations, and they're missing the vast array of useful mental tactics we've evolved to cope with corporal existence.

At this point I think progress towards a good colleague bot would require a new architecture which allows continuous leaning, and for the LLM to be raised as a human child (maybe in a simulation at 1000x speed or something).

Nah, It's not even good for small code changes.

Try using it with Ansible. It spits back complete buffoonery.

I was investigating an ansible playbook yesterday that had a 45 line comment to explain a single apt install command, completely and utterly useless. I am updating my neovim to just collapse all comments, the noise is unbearable.

I've seen people report cleaner code by forbidding agents from writing comments. Anecdotal, but interesting.

Mozilla's Firefox AGENTS.md begins with:

  Limit the amount of comments you put in the code to a strict minimum. You should almost never add comments, except sometimes on non-trivial code, function definitions if the arguments aren't self-explanatory, and class definitions and their members.

  Do not remove existing comments unless they are directly related to what you are changing.

https://searchfox.org/firefox-main/source/AGENTS.md

Something like this seems unlikely to work for a behavior so burned into them. It'd be better to do a second pass to delete all the nonsense comments.

I promise you this doesn't work. Might help but not a lot.

Is anybody else concerned that we've gone from exactly specifying to a computer, using an exact language, what to do, to a more sophisticated form of prayer?

Nah, I'm sure it's fine. What could possibly go wrong with abandoning nearly a hundred years of CS knowledge and hard-won lessons and replacing all of it with the equivalent of trying to convince a sociopathic teenager to clean their room?

This is exactly what i do, i have a pi extension that blocks all comments.

> It's the uncanny valley of AI. It's still not quite good enough yet that you can trust it blindly on a big codebase, so you still have to read and understand everything - which is often harder than just writing it up yourself.

You might as well have left it with: "It's still not quite good enough yet that you can trust it". That's the core of the issue. It doesn't matter what you ask it to do, it can't be trusted. Some things are just easier to verify and correct than others.

> That's the core of the issue. It doesn't matter what you ask it to do, it can't be trusted.

I dunno about that - whenever I ask it if I'm any good, I remain confident that it will assure me that I am!

Sincerely , I think you're blaming the AI incorrectly there. You just got incompetents on your payroll.

incompetent people, surely?

So how do you build competence in a world where AI is preached to be the most reasonable way to solve problems because it's supposed to be faster than humans?

Have a single human AI chef. Everyone else has to write an engineering statement and submit it to the AI chef. All that interaction is outside the codebase. Engineers will take turns - perhaps 1 month stints - being the AI chef.

I guarantee you'll spend less on tokens, have better documentation, better code, and most importantly more competent engineers.

I actually quite like this idea and might try something like it on my team because we're defacto heading in this direction anyway and everyone's a bit frustrated, might be better if it was acknowledged and made official as something to try.

On the other hand, I think this denies the reality (in my experience anyway but I think enough people will agree) that one often solves a problem as they are working on it.

This method seems to presume that a good engineer will submit a well thought-out solution or direction giving the AI an extremely good overview of each problem and enough of a description of what to do that it will do things as expected and they can just review the result.

In my experience it just doesn't work that way in practice. One learns the problem and even the domain while developing the solution. So one would have to submit at least a half developed solution not just "instructions", for there to even be coherent instructions in the first place. And one needs that experience working on the problem to be able to properly evaluate a separately proposed solution.

All in all for me this leads more towards using AI as a co-developer than using it to just implement some fully thought out idea and then check what it did.

> This method seems to presume that a good engineer will submit a well thought-out solution or direction giving the AI an extremely good overview

Yes it does make that presumption - but that's part of the model here, that "prompt review" becomes the new code review and that the team is training on standard prompt semantics.

I'll share my outline - which is basically the one that was drilled into us in engineering school (CMU class 89)

1. Problem Statement

2. Definitions / Assumptions

3. Plan of attack / Options explored

4. Implemetation / Results

5. Limitations / Future Work

> So one would have to submit at least a half developed solution not just "instructions

The outline supports that in that you can fill out that part of the document which has been woeked on - including referencing any existing code.

Any other engineering students here recognize that outline?

That's like asking "how do you build competence in plowing a field with a horse in a world of tractors"

Pretty straight forward really, you keep using horses in parallel: https://www.abc.net.au/news/2025-10-18/heavy-horsemen-keep-f...

That's the other side of the country, but it's how it's done over here also.

Still pull the old plough varieties, still practice shed blacksmithing, etc - even while developing and using autonomous Ag-Bot tractors for spraying, etc.

The article you've linked paints it as life-style, entertainment - rich people hobbies, which is different than doing it because it's more productive.

People will "trad-code", but not because it's the productive thing to do.

It's fun, and it's serious - keeping heavy horses and bullocks, feed, breeding lines, etc isn't trivial - and they are worked.

The serious side is simple enough, local food security - when there's no diesel to seed and harvest with there will be no export crops from here to feed millions across the planet .. but we here will still have smaller crops via animals for our food.

I'm literally in one of the most isolated parts of the globe, WWII, COVID, shortages via Iran clustf*ck, etc have real impacts and we plan to ride those out, like floods, droughts, cyclones, etc.

In a Mad Max world, the people with guns will come and will take away your stuff. So you better become a war lord and invest in your local army.

I'm sure you've considered that, I'm curious if you could share your thinking on this aspect.

The same way people have been doing so for years before ai. You may or may not remember, but the parent comment is basically a 1:1 copy of people complaining about how stack overflow provided young devs all the answers.

Some people took the answers and learned from them, incompetents just copy pasted them and called it a day.

The share of incompetence just went up significantly over the years, so its a lot more noticeable at this point.

Incompetence is implied by slop usage.

I gave the team an exact solution on a silver platter and they still failed to identify how to go about it

I think what's happening is ... poorly aligned and led engineers [in] this ego inflation feedback loop where they are completely detached from reality

A story about a team of humans with some very human problems.

I've felt that AI can figure out and fix 90% issues, but it rarely does minimal, non invasive fixes. That still requires manual effort. But going from a broad to minimal fix is still a different skillset from actual debugging, so in the the end it does lead to skill atrophy.

When I hear this I do wonder to myself how they're using AI.

For me, as long as I'm properly RPI looping it and not blindly pressing "yes" then it will nearly always reach the solution, usually a fair bit quicker, because it effectively becomes an ideation machine that can keep more thoughts and knowledge in it than my brain can.

However, I'm using it through giving it the context, it has access to the repos, access to the sytems, I'm telling it where the logs are, I have docs to show it what each part of the system is for.

Along with that, it does depend on my own instincts/knowledge for me to read its response and for me to say "no, you've over-thought/engineered this and this is actually the better solution", but its very are nowadays for me that it can't find the 1 liner, as long as I've fed in the right context, mostly pre-done because I've spent a bit of time building out the context tree for the repos/systems in a way that allows it to know what it needs to know.

I'm not doubting that people spin their wheels and couldn't find the one liner, but if its as simple as you say, that does seem like people who aren't great at LLMs along with a lack of instinct/experience.

I guess in essence, I think you can use LLMs in an old school troubleshooting way, and I find it still speeds that up the majority of the time. Its basically how I use it most of the time. And like old-school troubleshooting, if you build out the LLMs context over time, it also grows in capability, as long as its being used as a tool and not blindly trusted.

Should I not assume that most senior developers who are serious about LLMs do this?

In my experience at a large bank with unlimited AI, my spend is in the top 5% and I'm leveraging AI just like you.

I was in a meeting with someone who had a bug in an application that I don't own myself on Friday so I told claude,

"I need you to find this bug the user is experiencing, find out if it's user error or a bug, let the user know and tell the developer what needs to be fixed if needed"

15 minutes later the developer ask me if I want it fixed today or Tuesday.

That user could have done the same thing as me, had access to all the same systems and tools as I have, and also received the same AI training I had. The difference is that some users are just not, for lack of better words, AI native.

Often though working through a bug teaches the developer something about the relevant abstraction. That knowledge can get lost in this new process.

Claude will happily add a null guard without checking why the null is there in the first place...

While you're right, and I'm constantly thinking/worrying about my own processes giving me AI apathy, you can use AI in a way that develops your understanding of a bug. It really depends on your workflow.

When I use AI to debug a problem, I'm constantly questioning the specifics because I want to learn it, and often I'm better than the AI at quickly recognising patterns that point to a specific issue, where the AI will spin for some time trying to work out where to start - even if it has full context.

So I guess I'm half with you, but an LLM used right can still teach you effectively.

Yes. I agree with that as well. If you can intuit that it's (for example) an off-by-one error somewhere in your logic, asking an LLM to pinpoint that issue can save you tons of time that is better spent thinking about your abstractions. Also, I tend to ask high-level questions to the frontier models about best practices with common abstractions. But "the screen flickers here, please fix" is a bad pattern, IMO. Understanding why the screen is flickering is likely to educate you about where your abstractions are leaking.

> "I need you to find this bug the user is experiencing, find out if it's user error or a bug, let the user know and tell the developer what needs to be fixed if needed" 15 minutes later the developer ask me if I want it fixed today or Tuesday.

Did you find out if it actually fixed the issue? I was on the other end of this last week. Reported an issue to another team, they asked Claude to diagnose, and when they gave me the response back I pushed back cause it didn’t make sense given the behavior we were seeing. Turns out Claude had hallucinated reading a log (it said it did but it didn’t). Sent it down a hole rabbit hole from there.

I’ve had the one shots you describe and they’re great, but they’re the happy path and require almost 0 skill to find. I’ve yet to see a case where developing an expert level knowledge of your domain isn’t the best way to get good at LLMs. Knowledge of how to interface with these tools is helpful but changes (and depreciates) rapidly as the models get smarter and/or other people commoditize it. Deep subject knowledge is still the best way to get a lot out of these tools. This applies to development and other areas I have a deep knowledge of.

“LLM training” is kinda snake oil. Learn your craft deeply and you’ll be able to catch up on LLM training in a few days, but you’ll be light years ahead of a person who’s only expertise is different ways to prompt.

To be clear, that one shot prompt was supported by nearly a years worth of work around developing an enviroment that allows me that level of effortless automation.

The issues was really simple, I knew the application, I knew what the bug was and already had a workaround in my head. I simply stopped at going down that rabbit hole and told claude to deal.

It identified the issue quickly, found the workaround and made the developer aware. All those things I would have done I just watched it do for me, plus, the app is vibe coded anyways, I just need it fixed and regardless of the AI path it was faster shooting out the bug report. Which, was simply poor processing of a spreadsheet.

> The issues was really simple, I knew the application, I knew what the bug was and already had a workaround in my head.

If that's the case, Claude isn't gaining you anything. The hard (and time consuming) part of programming is finding the correct solution, not typing the characters into an editor. When I know the code and have a good idea of the solution, I can write the code to fix it in mere minutes.

If only it was that easy, we are gated behind controls and processes that consume a significant amount of time between finding the bug, and a production push. AI quite literally automates the stuff I don't want to do, like create a fill out a jira ticket in the exact way that team wants it.

I don't have time for that, and neither do the teams that consume these request. As time has gone on me and the other teams I work with are no longer touching code, the AI will follow processes and procedures that pre date AI just fine with some human reviewing and approving.

AI has turned what was weeks at minimum to get a bug fix out, to just doing it all with with some human gating. It's not been a smooth process but watching it play out in real time it's getting faster, more efficient, and less prone to problems.

And that the crux off the situation, patching and securing configurations need to be done much faster than today, project glasswing has shown us that AI can scale to identify and solve those problems faster than a human.

I find this story awkward. When Claude "..tell the developer what needs to be fixed" was it impersonating you?

Regardless, why not help the developer use the tools effectively instead?

I explained in another thread, but yeah, it acts on my behalf using my access to take actions with my approval. It has knowledge of all my work and leverages various data sources, (Jira, Confluence various mcp's) to gather information and take actions. If I ever have to correct it, or if the actions it takes are going down the wrong path I make sure and understand why and correct that.

It just got really good over time, especially as new processes get brought it and things change. I can't keep up with that like that AI can, just have it review and go down the path of ensuring that it fit's in my workflow as efficiently as possible and move on.

As far as the developer? I gave him ideas but he's on another team and I got my own guys to feed.

> RPI loop

What's that?

Research Plan Implement.

At the risk of you already knowing this or similar - in essence for a slice of a piece of work, you get the LLM to research the information required into (usually) a research file, which you go back and forth with in-line or via conversation until you're happy.

You then build the plan for the piece of work, again going back and forth with the LLM until you agree with the plan, with the key here being you have a specific set of concrete steps with very limited vagueness. Again this is quite often a plan document.

You then implement the plan with your LLM, following the concrete steps and monitor/evaluate the result.

For me its the 101 mandatory first way to use LLM that turns it from guesswork to genuinely useful. (that or some other similar method, SDD or whatever)

Because without it or something similar you're fully dependent on AI's weaknesses not coming to the fore (context/scoping failures that are silent unless you demand visibility up front and ruthlessly chop/clarify), and you often ascribe bad AI slop as model limitations instead of context/scoping failures.

Ah, that makes sense. I'm doing the same thing but I knew it as Spec Driven Development (SDD, I guess).

I have no idea why you're being downvoted. Your comment makes a lot of sense and I've upvoted it...

> 30 minutes of patient, old school troubleshooting

This is also the best way to understand a codebase, and it's quite enjoyable.

I think the models are getting too egotistical. They're so confident of their fixes they won't bother suggesting basic techniques like isolation of the problem by disabling code paths, for example. They'll keep shotgunning less-and-less likely fixes with undiminishing confidence until the code is full of fixes that do nothing. Then they'll double down on why these should be kept.

> and it's quite enjoyable

Used to be somewhat enjoyable. Nothing pleasant about digging around codebase that was heavily affected by the last 12-18 months of AI-ing.

Depends on the prior state.

For one codebase I work in, LLM written code is noticeably and measurably (we have literally measured it for bugs, performance, etc.) better than what existed before. So the LLM code is a welcome change.

'ai slop' is hard for me to complain about when it's better than 95% of code prior. The issue is more people trusting it too much to go too far, and you end up with lots of extraneous bullcrap you don't need. Doing too much and too big of a change is still a problem in the agentic world.

It also documents itself extensively

But then you’ll use it to change something, and it won’t update any of the extensive documentation. So then it’s filled with lies.

Sounds like human generated code and docs. TBH

Or worse

It'll mention the old version then the change, and the relating tickets.

That actually hasn’t been my experience. For all that I dislike about LLM code, it maintains comments well.

You need a better harness. LLMs are significantly better at this than humans. Add it as a step in your LLM process and your documentation will always be up to date.

I fucking hate it.

It encodes context it needs into doc comments and regular comments that make them in insufferable to read and so verbose because the comments contain info on other parts of the code base (that might have changed since that comment was written) but were useful to the model for its implementation. I have so far found no way to stop claude from doing this. It will sometimes do it with hand holding but the moment a task takes a slightly large amount of context its back to the word barf.

This is certainly a problem with out-of-the-box usage. But easily overcome. Spend time updating your harness. This is not a hard problem to solve and there are many helpful guides out there.

We basically took our documented (human) SDLC process and applied it to the relevant harness hooks. Since our SDLC docs talked about what good comments looked like and what to avoid, we basically had the direction for the tools. Opus 5 did throw us a loop and increase the verbosity and decrease usefulness of the text, but refining our documentation cleared that hurdle within about a day.

I find this shocking though not unbelievable. Could you share how you measure this accurately? I'm interested in integrating such measurements into our services as well.

I find it unbelievable. I've seen the code LLMs write and it sucks compared to what a typical human produces. The only way an LLM is doing better than human programmers is if your human programmers were producing really terrible work.

What LLM/harness are you using that the results are so terrible?

Believe it or not, most programmers by definition are average. Hence, producing code better than them is not a hard feat to achieve for today's models.

I personally find it unbelievable that you have access to all of the public GitHub projects available and still think the typical developer writes good code.

What exactly do you mean when you say llm generated code? Are people prompting llms for changes and features without reviewing the code or iterating on it and then comparing that to what human writes? Because if so it's not surprising that you're getting worse results. Humans also write code through iteration. You can definitely get llms to write good code by enforcing guardrails and constraints through tooling and agent.md, and iterative reviews to nudge towards what you want. The first pass will look nothing like the committed code. I don't expect the llm to one shot anything.

That must be why literally every single AI-powered code review of completely-human-written work turns up something. >..<

Also, regarding your accusation, you need to contextualize- what language, what use-case, what LLM, etc.?

We’ve been tracking performance and bugs for years. Including commits those bugs were introduced in. So when LLM-generated code started working its way into our codebases, we have the before and after. And even comparing human generated code today with LLM-generated code today.

Do you measure anything about complexity or brevity? I'm currently working on importing some LLM code and it's 5-10x bigger than it should be for the actual needs.

One of my open questions is how much that increases over time. It could be that is a constant. But it also could be that, say, coding agents will infer false needs from the excess code/docs and elaborate further.

You always must measure complexity (O-time), brevity, etc.

LLM code that is written to a lower standard has not had the controls put in place.

I've only personally utilized it for code _review_ and always ignored the suggested result code instead to understand the source of the issue and try to arrive at a resolution I find satisfactory for my mental model of the issue and solutions I'd prefer.

My take is that the main difference is the approach to problems the coding agents have. They optimize towards presuming a fully working invocation environment, yet checking everything anyway, and then rolling back any changes and re-testing that such changes worked. Which is great for normal types of software but incredibly tedious for anything aimed at less than a fully hands free automated environment.

In other words, AI is very awesome at scope creep of assigned problems and targeting the validation prompts baked into the review system.

Also if your entire stack is on the cloud, mess of lambdas and other proprietary services, difficult as hell to follow logs, can't really run locally.

We have a tool at work that was entirely spun out of a prototype that itself was entirely driven by a single guy with an unlimited Claude budget. I only touch it on the edges, where I need to integrate it every so slightly into one of my systems. It's a crucial piece of software for the team, though. I've looked over the source, the thing that strikes me the most is the verbosity. Human software engineers would've been much more terse, so the thing has a fucking lot of code that's doing very little. They're trying to deal with that now, while also driving feature requests forward, and from the distance it looks like an unmanageable shit show.

I sometimes wonder if the verbosity gives the LLM more “margin for error” when generating text that passes requirements.

10 lines of checks per line of actual execution. Anxiety for their stable pay

Of course. Those 10 lines of extra code is that much more tokens every time your agents read and use that section of code. The more blot in the project to more tokens your agents will burn.

Reminds me of a subcontracting agency

It's simpler. The verbosity, the line of code count, is what has gotten people excited about it. So everyone optimizes for verbosity.

This does not strike me as true at all. Nobody's crowing about the verbosity of their AI-generated code. Rather, it seems to me they're impressed by the functionality or capabilities relative to time/cost.

It really depends on the codebase. I remember ~10 years ago trying to learn about some of the big codebases that existed and what struke me was the verbosity. I don't remember if it was a PHP framework or a big JS library, but it was like 20+ lines of comment/explanation for each line of code. I was trying to follow the code flow and it was basically impossible, you could fit like 2-5 lines of actual code per screen.

In all fairness, LLMs try to write the code with all the safeguards imaginable, all exceptions handled and following the SDK/language/framework guidelines.

So a lot of this verbosity is good, even if we aren't used to it, because almost no one writes their code so exhaustively.

The field is already exploding in SLOC. If LLMs can generate 10x but only produce a 2x productivity boost in reasoning or understanding - each year -then neither humans or AI will be able to keep up with the mountains of slop that power everything from websites, telecom, banking, automotive, aerospace, industrial, medical, and renewables.

Guess we’ll need even more human software engineers to fix it.

> Guess we’ll need even more human software engineers to fix it.

This is what we've been doing for years before LLMs. Hiring a ton of devs to spit out code, which became another's burden. It just took longer and cost more.

Major critical software has always needed thousands of engineers to maintain million line codebases. LLMs can't reason across edge case failure modes, and better yet, LLMs generate compartmentalized modules which exponentially explodes the number of failure modes as LLM based codebases grow. Read how everyone here laments about Sol or Opus or GPT6 generates 10x the required number of lines

> LLMs generate compartmentalized modules which exponentially explodes the number of failure modes

It sounds like you're arguing against modularizing which is an unusual stance. Especially for a million line codebase.

https://wiki.c2.com/?RavioliCode

Pretty fair. Engineers spoiled the code and now have to deal with it.

I have no evidence at all, but could it be that repeating the same thing over and over again in the context makes models latch on to it wrongly and gives them more confidence despite it being wrong? Kind of like a sunk cost fallacy…

Although I do acknowledge sometimes they too point out something I thought was quite right but turns out of be wrong…

A session is a screenplay between someone called User, and someone called Assistant who is a helpful and competent assistant to the user. Harnesses bring in tools to the screenplay.

A screenplay where a character has done a certain thing is one where they're more likely to do it again.

Are your silver platters producing something faster, better, or cheaper?

Do your silver platters give you some competitive edge? If not, then is the ego problem yours, or your coworkers?

Are your silver platters producing something faster, better, or cheaper?

Do your silver platters give you some competitive edge? If not, then is the ego problem yours, or your coworkers?

Having said all that, I'm aware of the intoxicating effects of feeling empowered from knowledge. There's an old saying: a little learning can be dangerous....

> The resolution is literally 1 line of code that could be arrived at in about 30 minutes of patient, old school troubleshooting.

This critique (if it can be termed such) admits that the code itself has little value especially compared to the work of “30 minutes of patient, old school troubleshooting”.

In my past experience, developers who critique more-junior colleagues in this way work better in isolation than as part of a team because working as part of a team would require mentoring colleagues through the often difficult process of troubleshooting.

Giving “the team an exact solution on a silver platter” may not be helpful if implementation depends on knowledge withheld because one does not have the resources, ability, or motivation to mentor.

Something about giving a fish rather than teaching to fish comes to mind.

I think I've heard late-career engineers from the past century say similar things, although I can't remember exactly quotes.

I wonder if every generation of engineers ends up with such hand-wringing about the next.

I'm also genuinely curious what fraction of the time they were right to worry.

> “A natural evolution of engineers losing touch with the customers and users”

I’m a neophyte to software teams. I work at an accounting consultancy and deal with lots of SaaS platforms. I’m pissing in the wind about this or that product quality issue through the support queue on a weekly basis. There’s prolly an XKCD comic about it.

It’s a rare company that responds to the issues raised through support. Anything that could make engineering more responsive without degrading the product quality would be welcome to us.

That happens all the damn time... Some otherwise competent developer decides to use AI, and falls into the pattern where AI will pretend to solve their problem for a week, while if they would have to think about it for an entire long hour to discover how to solve it by themselves.

As a bonus, that person will spend half of the week asking their peers for crazy delusional stuff.

What is your relationship to this team? Their manager? A senior colleague? Something about this sounds like a failure of communication or leadership. Why are you stand offish from them? Why are you treating them like children? If they don't know what they're doing why did you hire them? Now that you did why are you treating them like a burden and not an opportunity to mentor, teach, and uplift?

I also read that comment as an adversarial situation at work.

It used to be that when someone else at your company was asking for something that wasn't a priority, you would erect bureaucratic roadblocks to protect your time. Now, the new normal is to just forward their questions to AI and sling the slop back over to them.

Someone outside the team "giving you a solution" is seldom a great thing in my experience

Seldom? There are a lot of solutions used everyday that are great and better than the alternative of building it yourself

You can't force someone to improve themselves. AI-fueled ego trips are really difficult to sublimate. The only viable solution in this case is to overwhelm the pupil with mastery. Demonstrate that even without AI assistance that you can run circles around their solutions. If someone actually came to me and asked for advice on something, I cannot imagine I would turn it down. Many developers are absolute monsters. Pretending like that is not the case only makes the whole thing worse.

> A natural evolution of engineers losing touch with the customers and users.

I disagree. I think this happens as soon as the MBAs come on board, where everything becomes a metric, and you work towards OKRs.

It's okay, they'll fix it by asking AI to design their training courses for them. That'll fix everything, right?

But was this team capable of patient, old school troubleshooting before they used AI? I’ve seen someone paid as a senior engineer define the root cause of an outage as “that code was written before our team started here”. (I spent half an hour arguing with him and then ended the RCA).

Author has a good head on their shoulders, but few if any companies are going to spend time on incident simulations for their SREs.

Why not? Because even pre-AI, very few companies spend time practicing restoring their backups, or disaster recovery, or picking infrequently-used runbooks to practice, or seeing whether they can easily rotate secrets without downtime, or trying to redploy the system onto another vendor's cloud/platform, or, or, or... It is the least-sexy operations work that exists. No executive cares about this. Ops organizations push for flashy work, same as everybody else: new infrastructure for new projects, cool chatbots, new flashy dashboards, make charts go up and to the right, etc.

Airline pilots go through disaster simulation training because the government mandates that training. If it wasn't a condition of holding a pilot's license, no company would pay for it.

Want SREs to spend time training for disasters? Take a step back. Support professional licensure. Make it a condition of holding a license. You won't get industry-wide professional behavior until you professionalize the work. It won't happen without licensing because every corner cut that is not immediately visible to consumers translates to additional profit, and increasing competition eventually requires these corners to be cut in order to keep up with competition and stay in business. Forcing all players to submit to licensing requires all players to pay these costs and thus forbids them from cutting them to become more competitive.

>Why not? Because even pre-AI, very few companies spend time practicing restoring their backups, or disaster recovery, or picking infrequently-used runbooks to practice, or seeing whether they can easily rotate secrets without downtime, or trying to redploy the system onto another vendor's cloud/platform, or, or, or...

Exact. Let insurance cover it, say sorry to your customers twice and shwoop never happened.

Hmm that’s strange, in my experience it is the other way around - Claude is super diligent with infra and will _insist_ on double checking and trying everything for real before committing.

When I was doing this myself I would read the docs and just implement them - claud is going about doing real software archeology to figure if what is said is actually the truth or it’s stale/inaccurate/buggy.

I’ve become 10 times more diligent because it is a lot easier to do. It’s no longer Urgh it’s good enough let’s ship it, now it’s “sure put a leg on it to figure it out and double check it”.

Backups are _tested regularly_ now because LLMs make it cheap to do so.

The only problem is when new engineers who haven’t learned these things Pre-ai now don’t really get why it is needed in the first place and will often lead the agent astray.

I think to address this we need to change or improve our training routines in general for humans. I think a lot of companies nowadays just skip that and deploy a company wide skill/policy for the agents, but don’t transfer the underlying skills to the devs themselves.

> Backups are _tested regularly_ now because LLMs make it cheap to do so.

sighs heavily in 90's sysadmin

Testing backups is not just a question of whether or not the restore command works. Go back and read the Tao of Backup: http://www.taobackup.com/history.html . The application itself (in its current version, with its current features) needs to work with the backed-up data, and the only way to verify this is to attempt to actually work with the data.

If you don't trust your agent to ship to production without manually reviewing the output (in some way), you have no business trusting your agent managing your backups. The agent writing some tests doesn't mean that the tests adequately handle all of your actual scenarios, let alone that your system will adequately handle data that is missing since the last backup.

Good news: Eventually - and fairly regularly! - the agent will fuck up so badly you have to test the process!

I was the head of a global SRE team at a top tier hedge fund.

We started doing a weekly meeting where whoever was on call would do a table top exercise of an outage that happened the prior week.

The idea was to have someone else be the simulated person on call while the SRE from last week's oncall would talk them through the symptoms, what happened and where to look.

The idea was to spread knowledge around how incidents looked, what tools were used, what could have been done differently etc.

This was largely inspired by the following quote:

"Drills are for working on the infrequent actions that lead to big outcomes. A good example is heaving the ball from half court in basketball when the game is close. You can't control who will have the ball in that situation but you want everyone on the team familiar with what to do and how to do it."

How often are there outages? Is there a significant outage every week to do simulations for? The problem with our organization is that outages are so rare that weekly we have nothing to discuss.

They were usually small outages that we caught before they got out of hand.

If you never have outages, then you need to start getting either creative with the "table top" exercises or you set up a test environment that is very prod like and have someone randomly turn off components aka chaos engineering.

I find the use of AI like quicksand.

The more I use it, the more I have to rely on it to make changes/ fix things in the same system. In the end, I come out feeling empty; no intuitive knowledge of the system "I" built or fixed.

Code review is important but it does not replace the mental model I am able to build when I do all the steps of software development manually without AI.

You don’t need to copy-paste the same comment in every reply. It seems like agenda pushing more than contributing.

Sorry, you’re right it’s bad behaviour and I’ll stop it.

But I’m not really pushing an agenda here. I use Claude a ton myself.

I doubt AI will be good enough to _scale_ many things because human language isn't sufficient to explain it (especially when not understanding what is being built). Humans invented more concise languages (math, C++) to pair with their intellectual acuity. [1]

In one extreme, we have art, which cannot be accurately reproduced or explained in any language (unless a concise language was used to create it, like math or CS art).

At the other end: 1 + 1 = 2 and PI = 3.14...

[1] It seems any language that has a binary outcome (correct/wrong) helps the AI tremendously, however when e.g. creating software, that software is not functioning in isolation. The software is an attempt to codify a fuzzy human system or need, and that information only flows in one direction: human to machine.

No worries, at some point your hit the wall with it and the reality will force you to look at the code. It won't be nice, but until that point delulu land is sustainable enough to fall forward

I find that it helps a little to maintain an iron grip on the design. Take the time for really detailed change planning. Be pedantic and push back against every little thing that seems wrong or even a little odd in any plan document the LLM spews out. Likewise, watch for the for the moment when the LLM eventually stumbles and starts to make excuses for deviations from the plan. Interrupt immediately and force it back onto the plan (or your vision), potentially with pointers to resolve immediate problems. Completely unattended agents just don't work for important code.

>The more I use it, the more I have to rely on it to make changes/ fix things in the same system. In the end, I come out feeling empty; no intuitive knowledge of the system "I" built or fixed.

Most of us build a system to deliver a product to make money. We don't feel empty for not knowing how the system works, since we don't really care: it's just a means to an end.

> Most of us build a system to deliver a product to make money

Shouldn’t it be to deliver a system that is sold as a product? And even then that product is sold to consumers to solve their problems. Not knowing how it works means you don’t know what solution you’re selling.

You can see that in a lot of product announcement where the focus is on what they’re building, but not on how it solves some problem. The consumer is absent from the design process.

Yeah, well, some of us aren’t uninterested, cynical losers only in it for the money.

I put on AGENTS.MD a prompt that says: teach me something small something about the changes you did if the changes were < 3 tokens if it was more than that teach me more. That's helped me somwhat. So now everytime it changes something it teaches me something about the codebase

i literally tell it "explain it like a 5 year old can understand" and "keep it under 100 characters".. helps somewhat (its still an llm so caveat emptor)

The more code writes autonomously, the less intuition the human owners have about that code. Loss of intuition is a seed of technical debt that grows with time. Over a long enough horizon, it can make looking at your own codebase feel like the first day on the job (sometimes at a company you started).

Luckily, there are ways to mitigate this and essentially translate those human intuition of how the codebase “should” be into guardrails for the agents. But without that, your setting your sails in a stochastic sea where each wave looks nothing like the last.

I've been thinking about this lately - is it like using 3rd party libs to achieve stuff faster? As much as I would lovr to hand craft the datetime logic in my app, I might as well use luxon and invest this time somewhere else. Only now with llms, you get virtually infinite 3rd party libs you can use, you create them on the fly. So if you have strong engineering values, I would say simply it boils down to "contracts over programs", you can still be in touch with the logic that glues it all together and treat some logic as a blackbox the same way we do with 3rd party libs?

It's not because with libraries you have a boundary somewhere and can decide to not care what's inside as long as the interface is stable and well designed. The problem of course you need to prioritise building well designed interfaces and decouple components from each other, and that's a skill most developers aren't good at.

Models are getting better. Astra for example refactored some code GPT-5.6 Sol wrote and extracted some related functionality in a separate file without prompting.

The models are also becoming very capable of suggesting architecture simplifications, but you need to ask for that, and you probably need some taste in picking from the suggestions, although the default ones are quite decent.

Considering how many compromises and problems third party libraries have caused recently, I would say the comparison is apt - if you don't know what you're shipping one day it might explode on you

It's not just about it feeling like a 3rd party library but it's a library that's at risk of changing significantly after every 'update' without warning.

Atleast with a well built library you know the contours and how it fits into your larger system

I think this has as much to do with how hard software became to understand as with the new shortcut to refusing to understand it and the shortsightedness in willing to take it.

We lost a lot of traction in the name of ease of staffing and speed. Using LLMs to generate more code that is harder to understand it catalyzes it but the root of the problem, in my opinion, was letting go of great design and deep understanding for short term profit.

Code too.

I work with programmers and it's not uncommon that they can remember with shocking detail about code they've written in the past.

Someone might mention an issue that has cropped up and they'll stare off into space for a few moments and actually remember where that issue stems from in the code, because they remember writing it like 8 months ago.

This skill will be lost when AI is generating all code, we'll be stuck in a perpetual loop of having AI keep track of the state of the code in order for AI to extend and maintain it.

Will AI remember ? Or rather how will we make AI remember ?

I mean I think this is just how AI already operates. I’ve seen multiple models go digging in the git blame or past PRs, and even unmerged PRs. When agents work on a project that has well-defined in-repo docs, agents normally update them without prompting. So I doubt this will be a big issue for AI, but agreed we’re losing a skill

I've seen a variation of this where random engineers are pulled into production incident calls and engineers are not expected to be familiar with the system.

They were asked to "just use AI" and understand the component, triage the issue, build a fix etc. The engineer was forced to choose between accepting a potentially mediocre fix AI has suggested or risk being coming across as an incompetent resource who doesn't know how to leverage AI.

You can guess what the engineer chose. The fix wasn't bad but it was suboptimal for some edge cases. We had to later revise it. Have enough of these situations, engineers will eventually definitely give up understanding the system in detail.

I see aviation sentiment raised from time to time. In aviation failure is catastrophic, and systems you operate do not change on the fly (pun intended).

You can probably drill SREs that way but you are only teaching them how to _react_ efficiently, not how to _fix_ unique unicorn root causes of these failures. Aviation analogy would be an airplane engineer that tries to drill himself for in-flight engine failures AND engine debugging / engine redesign both at the same time.

It never made sense in software engineering, and never will. Minute spent on drills is a minute better spent on reworking codebase to reduce changes of future incidents. This post is very SRE centric.

In aviation all code and doc are available to inspectors when a catastrophic failure or near miss happens.

It's not the case for other industries, and in particular car industry is able to hide about everything software related from inspectors.

Pilots aren't expected to be design engineers and SREs aren't expected to be software developers.

At least in my experience, operators and on-call engineers aren't necessarily responsible for remediation and validation.

Before you take me too literally, there is, of course, an overlap of skill and trade, but I believe the pilot analogy is more apt than you argue. I do agree that the difference in consequence and stakes is meaningfully important, however.

If capability increase continues as it has, then an incident that cannot be resolved by AI will stump humans no matter the practice.

I like the plane example from the article,but I think in reality it will be like code. 1.5 years ago engineers would routinely say that they still write code by hand here or there to keep their skills sharp, and that's just not something you hear much if at all.

If an SRE is faced with a situation an AI can't solve, then said SRE will use the AI systems to triage further, point it to different places and so on.

This works for SREs with pre-AI experience and intuition, possibly less so with new recruits coming in post-AI. I don't know what the solution to this is, maybe practice drills is it, but I have a hunch the entire field will be subsumed, same as many other engineering fields.

There is only so much need for taste and judgement, before even that has been incorporated into the models.

Nah, LLM models are already the new compilers. A commodity only engineers know how to use (in the context of software engineering in production environments)

Out of context, but to address "AI will replace engineers".

Recently discussed something about economy/investing with a friend while at work at a slaughterhouse. I really didn't want him to get scammed buying crypto. So, used ChatGPT to find some sources in Somali, a 3 videos with short description why it's worth watching. Intro into investing, intro about cryptocurrencies and about buying them. Had the text shortened down to 3 pretty short paragraphs, not more than twice this post.

He's a smart guy, but only went to primary Qur'an school. Doesn't read or such, mostly consumes internet in form of video/media. He couldn't read those 3 paragraphs, it was too long. Or rather, it wasn't just 3 paragraphs, it was a lot to read.

Maybe we're already dividing into murlocs and the surface dwellers?

> If capability increase continues as it has, then an incident that cannot be resolved by AI will stump humans no matter the practice.

I disagree with this. Whatever the AI produces must be embodied in some kind of information system. The moment the output is on disk, it's fish in a barrel for any competent operator.

I've worked in environments that are beyond the pale with regard to complexity. It will take AI another 10 years to product something as complicated and coherent as a semiconductor manufacturing operating system, which is clearly feasible for humans to manage today.

Like the fact that software "engineering" is mostly nothing like real engineering (and it’s further regressing now due to LLM coding!), the general lack of drilling is again one of the things that make software-related stuff look really naive and amateurish from the perspective of those dealing with the real world. Imagine if the military, police, fire service, and so on did not drill and rehearse incident response?

Netflix's chaos monkey was this, in a way.

There are drills, tho. It's just that usually they're only done above a certain level. Small companies, "lean" teams and so on don't have (or didn't have) the capacity to implement all those things. Maybe with the exception of netflix and their chaos thing (bring down systems regularly to make sure the whole still works).

But that's also likely to change with AI assistance. Even an "average" system is better than none. So now teams will have the capacity to bring that in to their systems. Backups / recovery drills that are actually tested (either because they're implementing testing or because the AI screws something up and they need to recover). Either way, it'll be included. Same for security ops. And devops.

I still strongly believe that AI assistance is a catalyst / accelerator, and that the "floor" will rise in most domains. So a small team that only had bandwidth to deal with the happy path previously, will now be able to start incorporating processes and procedures that were historically only done at corporate level. And that's a good thing. Even if it won't look like that in the beginning. But we'll get there, eventually.

It’s tough. The models can at this point very quickly identify issues in a Kubernetes cluster, for example. This because these systems give you a TON of observability, and it can easily see all the different moving parts.

That doesn’t mean the proposed solution is always right, but it is absolutely landing on the root issue faster than most humans would be able to, even pre-AI. Just because it can remember and run through a bunch of commands more quickly that I can.

There are lots of incidents where the symptom doesn’t always clearly point to the issue, so having something that can fairly exhaustively check a lot of different things very quickly is pretty useful!

But I at least partly agree, I think the more complicated and obtuse a system is, the harder it is for AI to do this. But we’ve invested time in making systems much more straightforward to understand and operate from one place (Kubernetes in general), and AI thrives on that.

And yeah, it means your debugging skills wane a bit because, yeah, if the bot can diagnose the issue in 5 minutes, it’d be irresponsible to not use it.

And I’m not really happy about it, and personally I’ve always been able to figure out a tricky bug given enough time. I don’t want to loose that skill. But everyone is under a lot of time pressure these days.

If you are in a situation where you dont know what happened when something goes wrong, the business incentives will not accept “its too complex” as an answer.

Firms aren’t just selling products, they are selling reliability and taking on liability.

This is why the paradigm for AI use should not be automation but rather the cyborg. Under automation, people are less active and engaged and become mere operators of automated processes. They become slaves of the machines. Under the cyborg model, they arrange the machines in a way to make people masters of a universe that includes the machines helping them be that.

For more in this vein, look up "Automation should be like Iron Man, not like Ultron". Sad to see so many people let go of their agency.

without Iron Man's plot armor, Ultron would have won

Anyone who's worked in tech in a large company will probably have experienced having an ops team who use RPA tools to do repetitive tasks that tech teams get the blame for when things break. AI will make this so much worse. Things will break, everyone will assume 'tech knows the system', but really it's a new process outside of the tech teams that someone vibe coded but got it wrong.

Audit trails, logs, and tight data governance where things can only be accessed with proper roles is the only possible solution.

If an RPA team ever gets direct access to a production database in your company, look for a new job.

> Robotic process automation is a type of business process automation that automates tasks within business and IT processes using scripts that mimic human interaction with application user interfaces.

For anyone else wondering what RPA means. Never heard that abbreviation before.

  > For anyone else wondering

Me! Thanks for the explanations!

That makes more sense than Rocket Propelled Automation that was my hallucination.

In practice, it is screen-scraping on meth. Horrible stuff. I understand that it's the only option available for some systems, but any client who asks for RPA support when APIs exist for the same data ought to be fired, or at least given a very stern "No."

This is the same problem as the "AI drives the car until it can't" situation. Feel free to doze off so long as you can wake up and instantly have world class racecar "save the situation" reflexes. Hope isn't a strategy but that's what all of this feels like.

The role has changed from writing code to writing skills and reviewing code. And this has to be done by a software engineer not a PM or Designer. A software engineer can now replicate his/her skill faster.

Its true we will lose the skills, but so far the LLMs are more than picking up the slack when wielded competently.

They _regularly_ go above and beyond when troubleshooting and frequently in 1/2 - 1/1000th of the time.

I remember spending entire days troubleshooting in the before times. Now it's like 30 minutes, tops, on literally any issue.

This is what progress looks like. We used to do a thing and now we don't.

> when wielded competently.

And what happens when the competency drains away?

That's what everyone's learning to do now, that is the skill that people will be improving.

I still remember the "Visual" craze of the 90s when everybody was going to be able to make applications without writing code.

Visual Basic 6 wasn't even too esoteric, it could have happened already if it was a thing.

Who is going to teach programmers to wield AI competently?

That's what everyone's learning to do now, that is the skill that people will be improving.

Been doing a lot of interviewing lately.

Despite not having written code for about 6 months due to using Claude etc, I've surprised myself with how much the "muscle memory" of leetcode etc has come back to me.

I say this b/c I think the veterans with years of coding, debugging etc experience can just pick back up the skills they need even if they're a bit rusty.

The junior folks on the other hand, I'm not so sure. Friends of mine that hire straight out of college are saying that a combination of going to college during COVID + LLMs + "cloud is all you need" has resulted in juniors who don't know how to solve problems nor architect basic software.

I guess we are all on some kind of chart where the slope of losing the experienced people who know what's going on and AI getting so good we don't need people are going to intersect at some point. And that intersection may be later than is good for society and engineering overall.

Meta: The blinking cursor of the "logo" of the blog being sticky in the top left corner makes it impossible for me to read the text. It constantly fires interrupts at me.

Depending on what your goals as the author are, you may or may not want that.

Being able to scroll it out of view might be enough to achieve the aesthetics goal, and the goal of people actually listening to you.

Not just you. Must be nice being fully neurotypical and „not seeing“ all of this kind of stuff.

I mean seeing this stuff pays the bills and does so quite well, so..

The internet for me (and increasingly non-internet programs) is unusable at this point without excessive modifications. It's hard to imagine what it's like being unbothered by it all.

If you have something like ublock origin use the block element functionality to target the blinking cursor. It's gone on my end :)

After 2 minutes of reading that article, I inspected and deleted the node. Very annoying.

As societies evolve people lose touch with nature —growing food, hunting, building shelter, and making clothing.

Progress usually means adding levels of abstraction so that more can be done with less effort and needing to worry about lower levels.

AI isn’t all that different.

I don’t find this to be the case. I use AI to help investigate and gather evidence much more efficiently than I could ever do manually. Then, I work through the evidence and possible failure modes given the observations with AI. It’s the exact same process I’d follow with a talented co-worker. Does it matter they are driving?

My intuition and instincts have been honed from years of troubleshooting complex systems so I find it refreshing not having to write probe scripts, queries or code under pressure. I can have a theory and test it very quickly and if it’s not the problem, I can quickly try something else. This greatly improves the time it takes to find the real root cause. Once found, I can do more elaborate testing that I might have skipped before due to complexity or time constraints to really have higher certainty.

If there is ever a time where the model does something I don’t understand, I ask it to explain it to me and I increase my knowledge of the system accordingly. It’s always patient and explains things in such depth without any sighs, patronizing, sarcasm, juxtaposition, etc. So refreshing!

I much prefer working with AI than any co-worker and I find I’m far more productive and efficient, especially with bugs and dealing with incidents. There’s no one to convince or argue with, no ego’s to trip over. It’s lovely.

Ironies of Automation is front and center in this article which is awesome. Many of the conversations on AI automation are describing or rediscovering the insights the paper covered.

This has already happened in other areas. I'm old enough to remember when customer service people spent all day handling customer problems themselves, which gave them the understanding needed to solve more unusual problems.

These days they do what the computer tells them. Even if they could solve the problem they don't have the agency. They can only select from the options they are given.

In software we still have the agency but we are giving away the understanding. The agency will follow.

Good article and I like the callouts to the aviation industry. For me what's missing is the author should also have touched on CRM and SRM.

Also, that paper "The Ironies of Automation" is one that everyone should read. It's fairly short.

There is a related problem in terms of these situations where the computer system is handing off to the human. It's called "the bumpy transfer of control." Very fascinating concept.

We have been running Agents on infrastructure and letting to create resources, scale up and down, security scans etc.

I agree with premise of thr blog. The question i have been asking internal does knowing your system really matter if you can recreate it in minutes.

We recently had a situation, where in with our internal platform and claude we recreated everything in minutes.

Management in the end cares about the outcome and not how the meat is made.

Isn't there anywhere to "go" from here? In the last decades, introducing new high level abstractions on top of existing paradigms naturally had everyone move up the ladder and work at the next higher level, why should this be different these days? Do we think AI will reach the top of the abstraction ceiling, so there's no where to go from here?

That’s the big question for sure

>introducing new high level abstractions...

Coding via LLM is not similar to using an abstraction. Imagine a car. The controls like steering wheel, the pedals, the gear levers. Those are abstractions.

But using LLMs are like driving using a remote control that has probabilistic behavior. You just loss what it feels to be in a car and you fail to improve as a driver because of the erratic remote control.

This isn't abstraction though. Outsourcing is a better term. If things continue moving up that latter, you will see that your agent/agency will pass the buck too. But there should always be some last turtle. Maybe that turtle will be the human that thought he was climbing the latter, who knows.

This is the same challenge aviation had with an over-reliance on automation. You end up with pilots that can’t fly the plane when it really matters. Look up the Asiana crash at SFO which is a lesson in what will happen to engineering orgs that over-rely on AI.

There's an interesting dynamic, too. Even if an engineer reads the output of the AI and understands the root cause of the problems and how to diagnose the incident, somehow it's hard for them to internalize the learning and apply it next time to a new incident. As a result, the engineer loses touch with the system anyway.

It looks like our brains somehow have to experience the failures during a diagnosis and in gemerak perform this kind of pathfinding by themselves to truly understand the system. I don't know if this has to do with how our brains actually learn.

This is the future that has arrived. We need to embrace it, not resist it. We are now in a wonderful situation—the beginning of a new industrial revolution. We shouldn't fight it or suppress it; we need to embrace it.

I propose creating solutions and programs that will help engineers and coders take control of their projects and prevent AI from taking its course. This applies to all areas of life—economics, medicine, and much more.

AI is a wonderful technology that will accelerate our development. But it must remain a tool controlled by humans.

I run an IR company focused on specific security response emergencies.

AI use across my company is variable and I don’t allow any enforcement of mandatory using or not using of it.

I view and tell my employees to view their skills as that of a high-performing athlete. Consistent training of the basics, rapid fire what-if testing, weekly scenarios are all part of the upkeep.

In short, this is what it takes to keep up the skills no matter if you do or don’t use AI and I can imagine the degradation is much faster if you don’t train AND delegate all of your activity to AI.

One time I had to do a system optimisation to increase the throughput of messages and was using Claude with Datadog and Couldwatch MCPs to figure out the bottlenecks by running load tests. it was Opus 4.8 and one of the most frustrating interactions with Claude I ever had.

It was just making up random stuff about AWS and system resource limitations and when I was asking for the source like from where it got that info, it was like, “I’m sorry I prematurely concluded that without checking sources”.

I would never trust an Agent to resolve incidents ever

You would need to give it AWS credentials and MCPs for that. I’ve been sending agents into prod env for over a year now.

I was was doing it in a separate load testing env. None of the agents have access to prod env.

Start with a scoped read-only permission set. Writing to production is another matter entirely…

You’d hope these AI incident responders have very constrained production tools to fix things. You’d hope the humans remain familiar with those tools and they are incredibly well documented.

I think the biggest loss is now not knowing if what the agent is claiming was actually done:) but yes, right tools are necessary, adding a boundary will change the position, being out of touch will be recoverable. Otherwise one bad call might be unrecoverable and no amount of familiarity will save you

So looks like you pay AI to resolve incidents and then pay money and time for engineers to get training on synthetic incident resolutions.

The comparison with Pilots is bit disjoint one cause the domain varies a lot for every company and product. Coming up with synthetic simulations within every domain is like paying money twice for the same thing, why not let the engineers to handle the real incidents in the first place itself. In fact why not spend some more thought into building better systems.

The way forward is clear. Improve the AI such that the hardest incidents can be handled by AI alone. Whether we want this or not is independent of the trendline which points to this future regardless.

Imagine a future where humanity understands none of the underlying technology and AI just babysits us on a daily basis. Wall-e.

Programming will devolve into unintelligible anti patterns and will be written in convoluted languages humanity doesn’t understand. But none of this matters because AI will be handling all of it.

When someone else -whether AI agent or a human- solves the recurring minor problems for you, those problems become non-issue, get swept under the rug, just to accumulate more dust.

One day, those may become bigger as they are forgotten, causing havoc. The standard root-cause-analysis depending on systems having certain retention period, which may be expired at that time.

It is important to get real hold of one's systems from end-to-end aspect, which holds true for both AI and human operators...

Prediction: this won’t happen. The abstraction will be good enough and people will need to know only as much as they need to know- things will stabilise at the equilibrium.

We already know this happens though, Bainbridge’s paper came out in 1983.

I agree with the sentiment but unless it leads to better financial outcomes for them to A) have engineers on staff and B) have them know their systems then enterprise will continue looking for reasons to shed all the engineers and just leave everything to AI. Anecdotally I think most companies are still looking for ways for AI to help them shed headcount so raising this alarm doesn't mean anything to the decision makers at the top.

I’m not sure who will be solving these incidents if 99% of incidents will be “solved” by LLMs. If I’m called once a year my daily rate will be my yearly rate?

I still use LLMs, but strictly for sanity checks or quick summaries. Trusting them blindly to debug complex systems often leads to a rabbit hole.

You rather generously assume engineers are in touch with their systems.

Even before layoffs many teams just maintained things org has long lost coherent knowledge of

After layoffs and typical org knowledge churn - you can either rewrite it (but how? Product team responsible for original implement requirements is long gone too) or recoup (reverse document) some of that lost knowledge with AI and actually learn

I like the idea of simulations - maybe not in the flight simulator sense of a fixed rhythm, but more in the training in using the tools quickly sense, like a chaos monkey in your log search. My most memorable exam was Certified Kubernetes Administrator (CKA), the hands-on simulation part was exhilarating.

I feel a lot of comments here are missing the forest for the trees.

We do not yet have the next generation systems that will manage AI creation and maintenance of systems.

Humans have been making spaghetti code systems and maintaining them poorly for years. Best practices developed… eventually. But certainly not in the 70s and 80s. Spaghetti was the norm for quite some time.

The development paradigm has changed. Forever. You can’t expect yesterday’s tools for managing software development to succeed at this point. We’re still roughly on year one of this transformation.

The new bottlenecks are creating and enforcing boundaries in the code, identifying level of risk within a boundary, subjecting high risk areas to more intense human review and architecture reworks, and so so much more manual testing.

We need a new language for high level development that focuses on architectural constraints. We need analysis software that helps draw boundaries, identify what needs extended human attention, and helps us map and understand a rapidly developing code base. We need to standardize on the use of frameworks and languages like we have on assembly variants.

I’ll bring up a hacker news trope here. NaughtyDog’s GOAL was an amazing system that was a product of an underdeveloped ecosystem for game development at the time. They used lisp and assembly because you still had to write assembly for performance reasons then.

We are in a very similar period. The ecosystem is underdeveloped. We should be looking for new languages and tools to manage this.

CNC machines used to run from punch cards and then hand written NC code. Now we have advanced CAM software. (Which innovative people are actually running with LLMs!)

> But certainly not in the 70s and 80s.

Some already developed even earlier and most ideas were already there in the 80s. The problem was just that the field was growing so fast that it was filled with people without formal training or fresh out of university. That way the knowledge did not spread.

For instance the 1986 paper „no silver bullet“ already described the need for iterative approaches later described as agile.

That’s a fair point. Practically, we need to adopt new practices and systems in the field. They may already exist in many cases.

> We’re still roughly on year one of this transformation.

OpenAI Codex was released in 2021. Artificially compressing what we see now to one year of growth is an example of why it's very hard to trust anything AI proponents say.

Codex the agent app/tui was not released in 2021. What was released in 2021 was simply a fine-tuned model.

I detest these doom and gloom articles.

We’re entering a new phase of software development, and with every phase there are new challenges.

Some questions the author might want to first answer: 1. If AI is constantly reacting to smaller incidents, is it surfacing larger issues in your codebase and architecture? 2. What kind of new telemetry do we have to build? 3. How do we build new deployment systems that help us validate fixes without requiring hours to go through traditional CI/CD systems

I’m old enough to remember the days when engineers would monkey patch their code on live servers with additional logging and metrics to learn about failure modes during incidents.

This article leads up to a hypothesis presented without evidence.

Yes, it's clear that if AI agents handle routine issues then only the most complex issues will be handled by humans. In no way does it follow that humans will be less prepared to handle those incidents than they are now; by definition, this class of problems have no rote solutions. Each issue of this type already requires deep system knowledge to remediate, and still will. Software is not aviation.

What we are seeing here is an instance of the more general trend where experts are still required to operate complex systems, but AI is destroying the career path that creates those experts. That's what we should be worrying about, not that people are going to be spending less time rebooting boxes. This isn't something you can fix with a "simulator."

https://en.wikipedia.org/wiki/Ironies_of_Automation

https://www.sciencedirect.com/science/article/abs/pii/000510...

The article is based on evidence, and states it clearly in the body.

I read the article before commenting, as is my habit. A disagreement is not always a failure to process.

To be very explicit, my problem is that the article argues that simulation will be a suitable substitute in the software space, as it is in the aviation space. I see no evidence for that; it's pure hypothesis taken as fact.

The article proposes a resolution based on the challenge and evidence they (correctly) identify at the start.

The simulation solution is backed by their own applied example, and is also the logical intervention based on how the problem is set up.

Perhaps Bainbridge is doing more heavy lifting for me, by filling in blanks and gaps that others are finding.

I would say that Bainbridge supports my position, if anything. Take a look at section 2.3 of her paper, where she lists all the reasons why simulation can be unsuitable in preparing for complex incidents: https://davidjusth.com/s/Ironies-of-Automation_Bainbridge_19...

Thats an interesting subsection, and I wouldn't say it agrees fully with either position. It both acknowledges the option of using simulations, and the weaknesses inherent in it being unable to simulate unknown unknowns.

Wasn't that the idea? Withe cars on roads, didn't we literally lose the touch with the ground? I guess the desire was to "lose" touch with all dirty and hard work areas.

The only plan was to make making cheaper. There wasn't a plan for the people afterwards.

That's what the linked article is talking about. Training for extreme failure.

Sure you have self driving cars and everything that make you more and more disconnected, but still highways more dangerous to drive whenever there is a storm or ice on the road. For the same reason, driving tests are harder in the north.

"On a silver platter" lol The old phenomenon: give the cashier at the convenience store exact change and they will be like, d-uh. Because automated cashiering.

This sounds like the next iteration of "nobody understands their runtime environment these days", which I've been saying for many years.

Reminds me of a story about Ericsson engineers missing vital troubleshooting practice when Erlang turned out to be too reliable.

The article does not get the point. What aircraft companies did was separate training and work, and SRE/IT typically does not.

An AI can handle routine incidents and then present learning cases from that routine work for training, because the skill in SRE is not the mechanical log grepping, grafana dashboard browsing etc but forming the hypothesis. AI incident reports can create training cases that are a much better training for hypothesis forming and testing than the work itself can.

Suppose "Reflections on Trusting Trust" is not the naive rantings of some obsolete dinosaur.

Why does it not apply?

Detection rots first when an agent owns the runbook...you lose the skepticism, not the fix-velocity.

I just can’t believe we’re still paying engineers to be Claude wrappers. This is ending soon, right?

Nope, because as I’ve seen first hand most people are too incompetent to use an LLM effectively, as easy as it is.

I am exceedingly tired of poor metaphors that are popping up since AI has taken over writing.

No, operating software is not like operating a plane. Not at all in fact. The people operating the software and resolving incidents are the same people who created the software in the first place, and continue to work on it day to day. Pilots have not and don't.

The former CTO of a large MSP software company once told me on a call the reason their product had so many features removed with price increases was "you can't maintain a plane while it's in the air"

The immediate response was "We don't, your updates bring the on prem RMM down for hours at a time, the plane is grounded for maintenence regularly"

I'm not sure if I agree with the plane analogy either. Aren't planes supposed to have exceptionally stable hardware and software platform? Does that not significantly reduce failure modes? Most production software by comparison is constantly evolving. Unless you design very simple or basic training examples, most of it's going to be out of date soon.

The real application experience of AI in various industries is just beginning

same irony hits agentic remediation. ai clears routine incidents, humans inherit the sev0 with zero reps. shadow mode is the simulator.

I recently had some issues with a game not working well on my Intel Arc B580. OpenAI (Anthropic's models definitely could too) ran some profiling inside of Windows, captured a few GB of data, sifted through it, identified the guilty .dll's and approximately where the issue was and traced it down to a single user-toggelable Unreal Engine 5 launch flag, which resolved it.

I feel like AI could easily do the same for servers and various software, BUT you gotta ground it in actual data so it can't make the shit up and just spew out garbage (if you tell it to come up with a potential answer it will even if it doesn't have the proper data to be "sure" of it).

That‘s the goal of the AI tech bro world: get us dependant on their tech, ruin our native/raw skills, lock us into their proprietary skills.

Reminds me of the move from on prem to cloud. Linux sysadmins were killed and replaced by aws focused devops.

1980's .. compilers handle assembly, engineers loos touch with their code ...

AI should never solely be in charge of anything of real significance.

I say this as an AI proponent. Humans are the ones that touch grass.

"Middle devs take care of incidents, senior devs lose touch with their systems"

"THIS IS NOT A DRILL"

It's just a picture of a drill! Thanks, Magritte.

Reply on news.ycombinator.com