Hacker News Reader

Comment by plaidfuji | Hacker News Reader

> Here’s the thing: most of our habits at work still assume that producing something means you have to understand it.

This has only ever been true in a relative sense. Until recently, one was expected to understand the software-level code they wrote. But not the machine code, or the computer internals at a deep level. You trust that those layers work. At some point in the past you couldn’t just trust that. At some point in the near future, you will just trust that the software code works. Instead you’ll just debate the spec (the prompt). Your thinking can move one level up the system hierarchy. Everyone’s can. We’ve done it before, we’ll do it again, and this won’t even be the last time it happens. Let go of the anxiety and let it happen.

View on HN

You trust machine code because it was produced by a deterministic system that was written and tested thoroughly by other engineers.

AI systems are not the same. You can't guarantee deterministic output.

Also, code is still the best language we have to convey our intentions as engineers into function. Natural language is too imprecise and ambiguous.

Understanding the code allows us to understand the intention behind the code and identify bugs, plan and architect better. Inspecting prompts gives you a blinkered view of the system meaning you're more likely to make incorrect assumptions leading to serious bugs.

Right, it's a chain of trust between the layers. The link I choose next to me helps determine the quality of successive links, whether I inspect them or not.

When I buy a wooden shack from a carpenter, I'd never expect them to core-sample every plank to ensure it's good wood. That's because a good carpenter habitually buys wood from a good supplier, and will have the experience to detect other suspicious indicators.

Similarly, my coworker might not have verified the machine code, but I can be pretty confident they didn't vibe-install a compiler by Ken Thompson that puts backdoors into the binary.

Are your teammates deterministic output machines?

In a sense yes. I trust that my coworkers will be accountable for their outcomes and accomplish them with the high quality bar I know they intrinsically hold themselves to. And that they understand our shared goals and if they don't, they will work to become aligned.

And if I can't say that for a coworker, well, that's performance feedback.

Highly likely, your trust is just based on your feelings, and they don't do much for the product quality.

Your coworkers are if anything less reliably deterministic than an AI. And the human version of a prompt injection is called social engineering and it’s far more of an issue than actual prompt injection and has been for an extremely long time.

I don't know what to say except that this has not been the case for me. Claude is a different person for every single prompt and has absolutely no core sense of what our goals are, or even who the "our" is that's having goals.

While my coworkers are very stable personalities of a consistent work drive, sense of pragmatism, things that they find more and less interesting, career aspirations, etc. they're not all the exact same, and I depend on that as I fit tasks and ownership to people.

Ok but why are you comparing different Claude sessions then? Of course they’re different, they have different experiences. Just because they have the same “name” doesn’t make them the same persona.

Compare a single session. It’ll be extremely consistent.

Prompt injection is often not the issue: it's prompt quality. If given a vague spec (prompt), the human is more likely to understand the problem domain, see what's missing, ask clarifying questions. AI is more likely to assume and take a probable path, which may not always be correct. It's (probably) not going to tell you no, what you're asking makes no sense.

At least the tools they used to use used to be so...Now the tools do this https://blog.coredump.cx/p/recursion-into-madness

But in chemical science you would not "understand" everything. You cook with ingredients where you don't fully understand then.

There are always limits.

Engineering human types are often guilty of saying something is low cost but then not doing it themselves. Great, "understand everything".

Trust us trust. You trust an entity. What the author is describing is a new kind of transaction. It's not that AI causes people to lie suddenly (although if you trusted the wrong people this might happen to you, but YOU were wrong then) but now you have a lot of work being done more at the edge of uncertainty and recommendations. Ideas are not ideas anymore, they are working implementations.

So instead of submitting discussions or requests for features, you literally submit implemented features because the cost is lower. This is good. You just have to realize that you are starting the discussion with a working implementation. If you continually try to act like this is not the case, without someone literally lying to you about it, this is more about just accepting a now workflow.

What about a team of human coders is more deterministic than an AI coder?

You trust that those layers work

Unlike AI, those layers were traditionally compiled using deterministic tools. And generally came with a suite of other tools to trace and debug them. I don't trust AI in the same way at all.

>This has only ever been true in a relative sense. Until recently, one was expected to understand the software-level code they wrote. But not the machine code, or the computer internals at a deep level.

Just because something wasn't required at all levels or 100% it doesn't mean it wasn't required or it wasn't important though.

I trusted that the machine code worked because it was a mostly mechanical transformation, with little to zero non-determinism involved, and the design decisions for the high level code were still mine and understood. Not the same with an LLM, even when see just as an added level of abstraction.

Given some LLM produced code, I also trust the compiler will produce working machine code matching it. Doesn't mean I trust the LLM code in the same way.

Then put your money where the mouth is and start committing prompts and having your LLM “compile” them with each production release.

I don't think that's the problem the source article is describing. I've seen a number of instances now where people produce work product they don't understand at any level. "Hey Claude, design this feature and send it out for review, thx!" will often seem to work if you don't know any better.

I'd argue the trust was always relative. Witness what happens when you are hired to untangle the buggy mess that was left by the previous "senior" engineer. Sure there was trust when they were in charge, but based on the code they wrote, they didn't actually understand anything they were doing.

Mostly agreed. In a previous life, I used to write code, then got Peter Principled upwards to the point where I "trusted" the engineers under me to do their job. Did I understand all the code my team wrote? Of course not. Even if I wanted to review all the code written by 20+ developers to the point where I could stay I understood it, there simply wasn't enough hours in the day to do so.

I figure what is happening is (human) software engineering as a discipline, at least on large scale projects, is rapidly and irreversibly mutating to software project management. But instead of managing teams of people, you're managing teams of agents. Where I disagree with your comment is in the assertion that everyone "can move one level up the system hierarchy." Some folks can't, so they'll stagnate. Welcome to the Peter Principle World.

> Even if I wanted to review all the code written by 20+ developers to the point where I could stay I understood it, there simply wasn't enough hours in the day to do so.

The difference of course being that you had 20+ developers who did understand it. With AI generated code no one has that understanding. The AI sure doesn't

> Some folks can't, so they'll stagnate.

And we have no answer for those people. Just "fuck you the world changed and you're being left behind"? Where exactly does that lead?

This is illogical. I see this everywhere but it is wrong.

If you're developing an agentiv system to build software for you in an automated way using a fleet of agents, you're still an engineer of a technical system even if that system is operating on somewhat fuzzy logic.

Management involves complete delegation and instead the task is to find a capable person for the task and then mostly let them just give feedback so you know whether you hired the right person or whether you need someone else. When you're working with a fleet of agents, you're not assembling a team of heterogeneous workers with different skills, you're hiring the same guy over and over again, hoping that he can do any job and yy same guy I also mean that the guy does not learn anything new over time.

Meaning traditional management skills count for nothing here.

because we could trust the layer below. Now we can't

Reply on news.ycombinator.com