AI Is Breaking This Thing We Call Trust | Hacker News Reader

AI Is Breaking This Thing We Call Trust

terriblesoftware.org

128 pointsmatheusml70 comments

I worry for the next generation.

You and I have a baseline. A brain that did most of it’s developmental phase while there was still some unified accepted version of truth (even if dubious)

Future generations will have to develop in a more splintered reality. A world where there is nothing to hold on to. Not even a dubious commonly accepted narrative

note this started years before generative AI, arguably around 2014 with the gamergate era

Earlier. I'd say it started right around the time SEO started taking off big time.

are you the same guy who made the fluid simulation thing and the little volumetric display on youtube?

Agreed. Truthiness started to wobble earlier, but I think it's fair to say it still existed. New era will shatter it into a thousand pieces

I think it started in 2000 when Al Gore who clearly won the election was discarded in favor of Dubya thanks to a biased Supreme Court.

I am not sure I believe in this "unified accepted version of the truth". Before the Internet it was a lot easier for the unified media [1] to convince people of whatever it wanted them convinced, but I find it a lot better for people to be exposed to opposing views. Who knows, maybe your country is the bad guy [2].

[1] https://pomf.space/MkiGm3hi.mp4

[2] https://youtu.be/3K1Z-8EvS74

The experience you envision does not sound so strange to me, having grown up in an intensely religious subculture which had a sense of truth quite distinct from that of mainstream society. Observing the blatant conflict between different communities' world-views engendered a healthy skepticism which has served me well through life. I think the kids will be all right.

we still have books. Hard truth. Hoard books.

> Here’s the thing: most of our habits at work still assume that producing something means you have to understand it.

This has only ever been true in a relative sense. Until recently, one was expected to understand the software-level code they wrote. But not the machine code, or the computer internals at a deep level. You trust that those layers work. At some point in the past you couldn’t just trust that. At some point in the near future, you will just trust that the software code works. Instead you’ll just debate the spec (the prompt). Your thinking can move one level up the system hierarchy. Everyone’s can. We’ve done it before, we’ll do it again, and this won’t even be the last time it happens. Let go of the anxiety and let it happen.

>This has only ever been true in a relative sense. Until recently, one was expected to understand the software-level code they wrote. But not the machine code, or the computer internals at a deep level.

Just because something wasn't required at all levels or 100% it doesn't mean it wasn't required or it wasn't important though.

I trusted that the machine code worked because it was a mostly mechanical transformation, with little to zero non-determinism involved, and the design decisions for the high level code were still mine and understood. Not the same with an LLM, even when see just as an added level of abstraction.

Given some LLM produced code, I also trust the compiler will produce working machine code matching it. Doesn't mean I trust the LLM code in the same way.

You trust machine code because it was produced by a deterministic system that was written and tested thoroughly by other engineers.

AI systems are not the same. You can't guarantee deterministic output.

Also, code is still the best language we have to convey our intentions as engineers into function. Natural language is too imprecise and ambiguous.

Understanding the code allows us to understand the intention behind the code and identify bugs, plan and architect better. Inspecting prompts gives you a blinkered view of the system meaning you're more likely to make incorrect assumptions leading to serious bugs.

Right, it's a chain of trust between the layers. The link I choose next to me helps determine the quality of successive links, whether I inspect them or not.

When I buy a wooden shack from a carpenter, I'd never expect them to core-sample every plank to ensure it's good wood. That's because a good carpenter habitually buys wood from a good supplier, and will have the experience to detect other suspicious indicators.

Similarly, my coworker might not have verified the machine code, but I can be pretty confident they didn't vibe-install a compiler by Ken Thompson that puts backdoors into the binary.

Are your teammates deterministic output machines?

In a sense yes. I trust that my coworkers will be accountable for their outcomes and accomplish them with the high quality bar I know they intrinsically hold themselves to. And that they understand our shared goals and if they don't, they will work to become aligned.

And if I can't say that for a coworker, well, that's performance feedback.

Highly likely, your trust is just based on your feelings, and they don't do much for the product quality.

Your coworkers are if anything less reliably deterministic than an AI. And the human version of a prompt injection is called social engineering and it’s far more of an issue than actual prompt injection and has been for an extremely long time.

I don't know what to say except that this has not been the case for me. Claude is a different person for every single prompt and has absolutely no core sense of what our goals are, or even who the "our" is that's having goals.

While my coworkers are very stable personalities of a consistent work drive, sense of pragmatism, things that they find more and less interesting, career aspirations, etc. they're not all the exact same, and I depend on that as I fit tasks and ownership to people.

Ok but why are you comparing different Claude sessions then? Of course they’re different, they have different experiences. Just because they have the same “name” doesn’t make them the same persona.

Compare a single session. It’ll be extremely consistent.

Prompt injection is often not the issue: it's prompt quality. If given a vague spec (prompt), the human is more likely to understand the problem domain, see what's missing, ask clarifying questions. AI is more likely to assume and take a probable path, which may not always be correct. It's (probably) not going to tell you no, what you're asking makes no sense.

At least the tools they used to use used to be so...Now the tools do this https://blog.coredump.cx/p/recursion-into-madness

But in chemical science you would not "understand" everything. You cook with ingredients where you don't fully understand then.

There are always limits.

Engineering human types are often guilty of saying something is low cost but then not doing it themselves. Great, "understand everything".

Trust us trust. You trust an entity. What the author is describing is a new kind of transaction. It's not that AI causes people to lie suddenly (although if you trusted the wrong people this might happen to you, but YOU were wrong then) but now you have a lot of work being done more at the edge of uncertainty and recommendations. Ideas are not ideas anymore, they are working implementations.

So instead of submitting discussions or requests for features, you literally submit implemented features because the cost is lower. This is good. You just have to realize that you are starting the discussion with a working implementation. If you continually try to act like this is not the case, without someone literally lying to you about it, this is more about just accepting a now workflow.

What about a team of human coders is more deterministic than an AI coder?

I'd argue the trust was always relative. Witness what happens when you are hired to untangle the buggy mess that was left by the previous "senior" engineer. Sure there was trust when they were in charge, but based on the code they wrote, they didn't actually understand anything they were doing.

You trust that those layers work

Unlike AI, those layers were traditionally compiled using deterministic tools. And generally came with a suite of other tools to trace and debug them. I don't trust AI in the same way at all.

Then put your money where the mouth is and start committing prompts and having your LLM “compile” them with each production release.

I don't think that's the problem the source article is describing. I've seen a number of instances now where people produce work product they don't understand at any level. "Hey Claude, design this feature and send it out for review, thx!" will often seem to work if you don't know any better.

Mostly agreed. In a previous life, I used to write code, then got Peter Principled upwards to the point where I "trusted" the engineers under me to do their job. Did I understand all the code my team wrote? Of course not. Even if I wanted to review all the code written by 20+ developers to the point where I could stay I understood it, there simply wasn't enough hours in the day to do so.

I figure what is happening is (human) software engineering as a discipline, at least on large scale projects, is rapidly and irreversibly mutating to software project management. But instead of managing teams of people, you're managing teams of agents. Where I disagree with your comment is in the assertion that everyone "can move one level up the system hierarchy." Some folks can't, so they'll stagnate. Welcome to the Peter Principle World.

> Even if I wanted to review all the code written by 20+ developers to the point where I could stay I understood it, there simply wasn't enough hours in the day to do so.

The difference of course being that you had 20+ developers who did understand it. With AI generated code no one has that understanding. The AI sure doesn't

> Some folks can't, so they'll stagnate.

And we have no answer for those people. Just "fuck you the world changed and you're being left behind"? Where exactly does that lead?

This is illogical. I see this everywhere but it is wrong.

If you're developing an agentiv system to build software for you in an automated way using a fleet of agents, you're still an engineer of a technical system even if that system is operating on somewhat fuzzy logic.

Management involves complete delegation and instead the task is to find a capable person for the task and then mostly let them just give feedback so you know whether you hired the right person or whether you need someone else. When you're working with a fleet of agents, you're not assembling a team of heterogeneous workers with different skills, you're hiring the same guy over and over again, hoping that he can do any job and yy same guy I also mean that the guy does not learn anything new over time.

Meaning traditional management skills count for nothing here.

because we could trust the layer below. Now we can't

I wonder when the people will get that intelligence is not only directed at the external, but only really starts when you look at the internal (joy, pleasure, traumas, taboos, awkwardness, abuse, trust etc.). Look up the word "interoception".

The elite broke trust.

No one made people like Sam Altman into a compulsive liar. No one made Musk call a service full self driving, and then make it return liability to the user the second it decides an accident is about to happen.

Stop removing agency from the people making the top level decisions and reaping all the rewards.

Would you claim that someone who took 5 shots in a row and drove home wasn’t responsible for an accident because the car was what physically did the damage?

There were always a spectrum of people to choose from to become elites. We chose a system that chose the ones who compulsively lied.

To be fair, it has probably only been that way forever. At least we live in a time where some people might realize it.

That choice was also made within a framework of another system. It's systems all the way down.

You are removing agency from everyone who is not visible, even though they might buy trash and make it profitable for thousands of sellers on Amazon to lie, and to fraudsters to lie, etc.. Pointing fingers at those who are the most visible and happened to be richer than you is something that people have always done as it is so convenient. It becomes even more convenient if those people have money, so you can turn the punishment into robbery. Doing good and distributing wealth at the same time. How convenient! Greetings from Germany.

And by the way, I would really love to somewhere find a log file of your personal life and see how often you have promised something, and it didn’t turn out quite to be as perfect as you thought, or took longer than you thought. Or were you perhaps thought that you had achieved a goal when somebody from the outside might disagree. Difference between you and someone in the spotlight is that for them such a log exists; for you it doesn’t.

Oh, I thought the main difference was I wasn’t given outsized rewards, wealth beyond imagination, power over society, and most importantly.

I actually have consequences when I make promises and fail them.

I don’t get to keep claiming full self driving will be done next year for a decade and then move on to the Nasdaq changing its rules to get my next company fast tracked into everyone’s 401ks.

I don’t get to commit felony after felony and then do infinite appeals to never pay my fines and move into the White House.

You are making a ridiculous comparison but I guess keep defending the people fucking us all over if you want.

The assumptions we made about code always led to the biggest issues. If you assumed a junior engineer, or even a senior engineer not familiar with the local practices, would not make certain mistakes, then those mistakes always slipped through. If anything, the AI makes us be _better_ reviewers, since we need to check more of our assumptions at the door. I used to check the compiler assembly output, in case I could optimize it either by hand, or by changing my code to use a do-while instead of a for loop. I don't do that anymore. The same thing is happening with AI-generated code. I used to have to check it all very carefully, and regenerate code when the AI chose to add a whole new mechanism to the code base rather than re-using an existing mechanism, to keep the code solid. That happens less and less, and soon, it won't make sense to read over the code at all. Maybe in some specific circumstances that will be necessary, but less and less overall. And that's ok. We have moved to higher and higher levels of abstraction over the decades that I've been coding, and each time we can take on bigger problems and invent new things.

No not AI. People who believe in it. But the main purpose of AI is to give answers. Not good/right answers, but just answers. That's why it is used to manufacture consent.

> AI Is Breaking This Thing We Call Trust

An increasingly large percentage of people being lying POS grifters is breaking trust.

AI just happens to be a tool that the laziest of that group of people leans on a lot to supercharge their grifts.

But our massive and growing trust problem predates and is way deeper than AI.

Yeah, I think what they really mean is people, especially those with power, broke this thing called trust and now they're trying to use """ai""" as a scapegoat.

It'd be along the same lines as saying "I was drunk driving 75km/hr over the limit down a busy suburban road during the middle of the day and 5 people died because of it so we CAN NOT TRUST CARS"

And everyone stands up and claps

Both, in a way. The car that can drive 75km/hr existing enabled the person to choose to use it while drunk. As in, there was no trust beforehand, but the means used to "break" it were not as accessible.

I think one way to approach this is by deliberately choosing less capable tools (but still AI-assisted). You don't have to delegate huge tasks to agents wholesale, but it requires fighting against mainstream harnesses to do so because they exist to sell you more tokens. Model providers are not incentivised to offer less autonomous flows.

I wrote about this frustration a few months ago [1], then realised that if the model providers won't do it perhaps I should try fixing it myself. So, shameless plug klaxon, I wrote an open-source harness that tries to work more like pair programming and less like code review [2].

As usual, my "Show HN" post [3] sank like a stone, so I think perhaps most engineers aren't really interested in this direction. It is possible though, we don't have to just accept whatever tooling the labs throw at us.

[1]: https://philbooth.me/blog/agentic-coding-and-mental-models

[2]: https://www.opairdev.org/

[3]: https://news.ycombinator.com/item?id=49625009

Have you tried `pi`? What do you think of it?

I have. I thought it was okay, but it wasn't really the radical departure from other coding harnesses that I was looking for. Perhaps I could have made it that with more customization, I'm not sure.

My own harness doesn't provide the agent with shell access, so everything has to go through focused tools with limited functionality. There's also a navigator mode where the agent has no writable tools at all, instead it monitors the project for changes you make in your regular editor and comments on them in real-time. That stuff seemed easier to implement from the ground up.

AI is not breaking trust. People using AI are breaking our trust.

Every generation has had tech doomers. Every single one. History repeats. Yet we collectively move forward. This time it will be no different. These folks are wasting time when they could be doing something productive.

plenty of ex-civs had techdoomers who were right. When tevh doomers have no salient points to make, its fair to not listen, but this time around i think something is different to historic doomerism.

Trust is a fundamental thing, and i cannot fathom how a post-trust society could possibly host life as we know it. Life ad we know it comes from systemic efficiencies that are rooted in trust, requiring it deeply. no trust means extreme loss in system efficiency, in a system already creaking under the straign of supporting current populations.

post trust is in my reakoning a serious issue to engage with.

There are plenty of low-trust societies. They have things like armed security guards and barbed wire fences outside every apartment complex. Coughsouthafrica

right, but that surive in isolation, heard immunity to collapse so to speak, but globe wide? pandemonium. we're back to fiefdoms and warring city-states in a generation.

> There are plenty of low-trust societies

And high trust societies have spent the past two decades importing tons of H1B style workers from many of the lower trust countries

Which ones are those? Sweden?

Be snotty about it if you want but I didn't invent the term "low trust society", it's a pretty accepted term in social psychology from what I can tell

And based on that, countries like India, Pakistan, Phillipines, Nigeria, Syria all quality as "low to very low trust", and they also represent some of the highest numbers of immigration to many countries right now. A full 30% of immigrants to Canada over the past 5 years are from India, for example.

I don't think it is that unreasonable of me to suggest that people from low trust societies bring low trust behavior with them.

Which does not imply they are untrustworthy people by the way. It does imply a different set of expectations and public behaviors though, which often chafe with their new country

Isn’t “low trust society” just the latest racist dog whistle though? When you say ‘low-trust society’ don’t you just mean the third world, populated by brown people? Or are there examples of first world white low trust societies?

No, I mean low-trust societies. I don't know if a low-trust society can be first-world because it prevents value creation.

Actually, it seems that every time we do move forward, we do make things worse. So I think the tech doomers actually have a point.

I would challenge the "move forward" narrative. Move forward as in time perhaps, but on all other directions we are walking backwards it seems

I agree with that!

Reply on news.ycombinator.com