Hacker News Reader

Comment by nradov | Hacker News Reader

LLMs are non-deterministic just like humans and so security can be handled in much the same way. Use role-based access control to limit access to the minimum necessary to do their jobs and have an approval process for anything potentially risky or expensive. In any prominent organization dealing with technology, infrastructure, defense, or finance we have to assume that some of our co-workers are operatives working for foreign nation states like Russia / China / Israel / North Korea so it's the same basic threat model.

View on HN

LLMs are deterministic*. They are unpredictable or maybe chaotic.

If you say "What's the capital of France?" is might answer "Paris". But if you say "What is the capital of france" it might say "Prague".

The fact that it gives a certain answer for some input doesn't guarantee it will behave the same for an input with some irrelevant (from ja human perspective) difference.

This makes them challenging to train and validate robustly because it's hard to predict all the ways they break. It's a training & validation data issue though, as opposed to some idea of just random behavior that people tend to ascribe to AI.

* I know various implementation details and nonzero temperature generally make their output nondeterministic, but that doesn't change my central point, nor is it what people are thinking of when they say LLMs are nondeterministic. Importantly, you could make llm output deterministically reproducible and it wouldn't change the robustness issue that people are usually confusing with non determinism.

You are technically correct but that's irrelevant from a security perspective. For security as a practical matter we have to treat LLMs as non-deterministic. The same principle applies to any software that hasn't been formally verified but we usually just gloss over this and accept the risk.

Non-determinism has nothing to do with security, you should use a different word if you want to talk about something else

This is pedantry, temperature introduces a degree of randomness (same input different output) to LLM, even outside of that non-deterministic in a security context is generally understood. Words have different meanings depending on the context in which they are used.

Let's not reduce every discussion to semantics, and afford the poster a degree of understanding.

If you're saying that "non-determinism" is a term of art in the field of security, meaning something different than the ordinary meaning, I wasn't aware of that at least. Do you have a source? I searched for uses and found https://crypto.stackexchange.com/questions/95890/necessity-o... and https://medium.com/p/641f061184f9 and these seem to both use the ordinary meaning of the term. Note that an LLM with temperature fixed to zero has the same security risks as one that doesn't, so I don't understand what the poster is trying to say by "we have to treat LLMs as non-deterministic".

I understand the point that you are making, but the example is only valid with temperature=0.

Altering the temperature parameter introduces randomness by sampling from the probability distribution of possible next tokens rather than always choosing the most likely one. This means the same input can produce different outputs across multiple runs.

So no, not deterministic unless we are being pedantic.

> So no, not deterministic unless we are being pedantic.

and not even then as floating point arithmetic is non-associative

When processing multiple prompts simultaneously (that is, the typical use case under load), LLMs are nondeterministic, even with a specific seed and zero temperature, due to floating point errors.

See https://news.ycombinator.com/item?id=45200925

This is very interesting, thanks!

> While this hypothesis is not entirely wrong, it doesn’t reveal the full picture. For example, even on a GPU, running the same matrix multiplication on the same data repeatedly will always provide bitwise equal results. We’re definitely using floating-point numbers. And our GPU definitely has a lot of concurrency. Why don’t we see nondeterminism in this test?

Humans and LLMs are deterministic in the sense that if you would rewind the universe, everything would happen the same way again. But both humans and LLMs have hidden variables that make them unpredictable to an outside observer.

Humans and LLMs are non-deterministic in very different ways. We have thousands of years of history with trying to determine which humans are trustworthy and we’ve gotten quite good at it. Not only do we lack that experience with AI, but each generation can be very different in fundamental ways.

We're really not very good at determining which humans are trustworthy. Most people barely do better than a coin flip at detecting lies.

Determining trustworthiness of LLM responses is like determining who's the most trustworthy person in a room full of sociopaths.

I'd rather play "2 truths and a lie" with a human rather than a LLM any day of the week. So many more cues to look for with humans.

Big problem with LLMs is if you try and play 2 truths and a lie, you might just get 3 truths. Or 3 lies.

Lies, or bullshit? I mean, a guessing game like "how many marbles" is a context that allows for easy lying, but "I wasn't even in town on the night of the murder" is harder work. It sounds like you're refering to some study of the marbles variety, and not a test of smooth-talking, the LLM forte.

The biggest difference on this front between a human and an LLM is accountability.

You can hold a human accountable for their actions. If they consistently fall for phishing attacks you can train or even fire them. You can apply peer pressure. You can grant them additional privileges once they prove themselves.

You can't hold an AI system accountable for anything.

Recently, I've kind of been wondering if this is going to turn out to be LLM codegen's Achilles heal.

Imagine some sort of code component of critical infrastructure that costs the company millions per hour when it goes down and it turns out the entire team is just a thin wrapper for an LLM. Infra goes down in a way the LLM can't fix and now what would have been a few late nights is several months to spin up a new team.

Sure you can hold the team accountable by firing them. However this is a threat to someone with actual technical know how because their reputation is damaged. They got fired doing such and such so can we trust them to do it here.

For the person who LLM faked it, they just need to find another domain where their reputation won't follow them to also fake their way through until the next catastrophe.

This is a fascinating idea, imagine a company spins up a super complex stack using llms that works, becomes vital. It breaks occasionally, they use a combination of llms, hope and prayer to keep the now vital system up and running. The system hits a limit, say data, code optimization, or number of users, and the llm isn’t able to solve the issue this time. They try to bring in a competent engineer or team of engineers but no one who could fix it is willing to take it on.

You can hold the person (or corporate person) who owns or used the LLM accountable for its actions. It's like how dogs aren't really accountable. But if you let your dog run loose and it mauls a toddler to death then you'll probably be sued. Same thing.

(Yes, I am aware this isn't a perfect analogy because a dangerous dog can be seized and destroyed. But that's an administrative procedure and really not the same as holding a person morally or financially accountable.)

Your source must have been citing a very controlled environment. In actuality, lies almost always become apparent over time, and general mendaciousness is something most people can sense from face and body alone.

Yeah, so many scammers exist because most people are susceptible to at least some of them some of the time.

Also, pick your least favorite presidential candidate. They got about 50% of the vote.

I think most neutral, intelligent users rightly assume AI to be untrustworthy by its nature.

The problem is there aren't many of those in the wild. Only a subset are intelligent, and lots of those have hitched their wagons to the AI hype train..

Even with a very charitable of you to LLM document-building results, these "versus a human employee" comparisons tend to ignore important differences in scale/rate, timing security, and oversight structures.

Reply on news.ycombinator.com