However an AI agent, or the model powering it is stochastic by design. How can you certify something which doesn't behave the same twice, and more importantly we don't understand how it works 100%?
BTW, really, how is that AI observability work is going in the frontier labs? Do they care, even?
I also agree that qualitatively, this technology seems different than the others. However, I feel that people tend to overly fixate on their internal stochasticity. Even if LLMs' internal mechanism is nondeterministic, shouldn't we be able to verify their "side effects" aren't harmful? Of course, "harm" is subjective and at this scale, the most effective way to verify behavior is probably some kind of LLM-as-judge...
Anyway, in this case the problems have occurred while actually running the evals themselves, so again, we're in a situation where we can't even confidently test these things and know that they won't cause harm in the outside world.
That's a question any lawmaker has already had to ask about technology all the time.
I'm not saying they came up with great answers, but there's nothing qualitatively new about that.
The stochastic factor doesn't change the fact that companies have to be accountable for the harms their software causes. That's just basic liability law.
We're on the same page. What I'm saying that certifying them as safe is harder than certifying a drill as safe, and we shall be more cautious about AI related technology and be more stringent about the can of worms it opens without hesitation.
Many physical machines and components come with a datasheet that will list their tolerances.
Failure to correctly document tolerances does in fact get you sued.
However, while this is truly a great idea, we're not going to be able to make it work for computational systems. Computers, software, and also LLMs are sensitive to initial conditions. Which is why tolerances are not so familiar to computer people. (but not entirely: eg your PSU might list 110-240Vac/300W as input tolerance)
Interestingly, LLMs actually have a somewhat lower sensitivity to initial conditions than traditional interpreters. See what happens if you misspell "What is One Plus nOe?". So they're actually a skosh off the edge and towards the middle, though I'd argue still very much at the computational end, just from the sheer scale of the valid inputs and outputs.
Mind you, if you have a pretrained LLM doing a measurable task on a line, possibly some sort of tolerances could be determined. Not so much when doing arbitrary chat.
Something unintuitive: I bet that often setting the temperature > 0 (aka introduce stochasticity deliberately, variously comparable to dithering or simulated annealing in other disciplines - doing the thing where you escape local minima) will tighten the output tolerance range and improve reliability, especially in iterated processes. This works for a lot of physical and digital processes actually, and LLMs simply stole the same trick.
(edit: I'm trying to compress a huge chunk of dynamics intuition in a few lines here. Hopefully still useful.
TL:DR; Everything real is continuous and noisy if you look close; and you're really trying to build attractors and bound variance, if you can. )
By verifying that all of its possible behaviors conform with the "it works" spec, regardless of which of those behaviors it chooses.
Monitoring with a known-safe fallback is the easiest case.
Casinos can't make slot machines that literally never pay out, but it's a different result every time you pull the lever. We have existing legal frameworks for how to regulate things that aren't perfectly predictable (an economist might argue that if it were possible to predict slot machines then casinos with them would all go out of business).
Liability will shift to the maker of the tool if they claim that it’s easy to use, safe, or that you don’t need unique skills or training to use it.
That would be considered reckless.
Cars analogy - We have licenses for cars, and different types for different vehicle classes.
Cars have to be rigorously tested to meet standards to be considered road safe.
For example, for a runaway car (example from a sibling comment), the driver could be liable because they forgot the parking brake. The driver could be liable for a lack of maintenance and inspection. A mechanic could be liable for not reinstalling brake pads correctly. Or the manufacturer of the car or the brake pads could be liable because of a systemic defect.
Or it could grow even more complex, maybe the brakes are designed that they have to be maintained in a very specific way, and the mechanic did a reasonable maintenance and inspection but it failed later due to this maintenance. That could split liability between the manufacturer and the mechanic.
As an example, with other software, you as a developer or operator of a software have a duty to ensure it does not access computer systems you do not own in unintended ways. And this could go beyond liability into criminal territory.
It'll be interesting what OpenAI gets slapped with there.
Is AI less deterministic than an airline dealing with weather?
Of course not. The difference is one of those two things has a culture of safety and is well regulated, and the other one isn't.
For that thing, procedures and regulations are built. So regulations fit into a well understood phenomena, incl. "return back because that thing is way powerful for us".
For the same prompt, an AI model can return two completely different outputs, incl. but not limited to content, length, formatting and tiny details. What you get is a single instance. So, regulating an AI model for safety or any other property is not as easy as regulating air travel. Moreover, you have much stronger motivations for regulating airlines. Otherwise people die in a visible and gruesome way.
With AI, it's easy to whitewash problems. Somebody committed suicide? "They were already unstable". AI told something wrong and created problems? "The tech can’t guarantee truth because it's not alive, it can't understand right and wrong". It did something good? "It's probably a sentient being, we shall respect them".
I'm for regulating these things. They are dangerous as they are useful (sometimes), but the forces and motivations for regulating it is not the same.
What it's not is God or an independently conscious entity that somehow trumps a thousand years of common law that's built up until now about torts and liability.
Of course, there are some novel issues here that'll pop up here and there, but the idea that this is fundamentally different is propaganda on the part of these AI labs because the more boring, obvious situation doesn't favor them.
Agreed, the tendency of people on tech to assume that whatever the most recent thing we've come up with is unprecedented and shouldn't have to follow all of the established patterns we've built up in society for making things safe is wild. I don't know what the next Big Thing will be but I'm pretty confident there will be people claiming it's so different from everything before that we have no choice but to throw out all of the rules for it in the name of progress.
Yes, obviously? The responses of an airline to inclemement weather fit in a reasonably small set of responses, mostly involving rescheduling and/or rerouting flights.
The current AI predictability would be like if some airlines decided to do 9/11 when it was raining.
The current so-called scandals about AI hacking into other companies were because a bunch of human beings intentionally configured the software to go and do exactly that thing.
There's nothing deterministic about weather, so hopefully you're not just being disingenuous.
It's obvious that the global transportation system, or financial markets, or any number of other things are complex adaptive dynamic systems that are on par with AI in terms of their emergent properties.
Check my username. It's a concept I spent a lot of my life paying attention to.
Just because something has elements of autonomy or is adaptive doesn't make it particularly novel. We've dealt with those kinds of systems for centuries. The solution is to make rules and enforce those rules by whatever means are needed to meet the specifics of the case.
The rules, of course, are enforced against human beings.
I mean one of the outcomes of an airline was 9/11.
That's sort of my point. Complex systems have emergent behavior. That's always been true. The combination of AI and humans and packet switched networks is a complex system and we've seen most of the issues created by this already and have tools for dealing with them. Obviously with some genuine novel issues likely to come, much in the way that 9/11 would have been less possible using ocean liners.
I'll stick to my original point though. AI absolutely IS deterministic. If you run an AI algorithm on a microchip, literally nothing of note will happen in the human world. Some transistors will change state. It's ONLY when it is integrated into a complex human system that it gets interesting.
Just like lots of other things.
You're the one being disingenuous. Look at what you wrote.
> Is AI less deterministic than an airline dealing with weather?
You didn't talk about how deterministic the weather is. You talked about how an airline responds to a weather event in comparison to AI, which means that it's about responding to presented information by making a decision.
The state of the weather does affect your micro decisions, but the rules you follow are the same every time and the rules are as deterministic as possible even if they rely on pilot intuition.
I think you are profoundly confused.
An airplane, and an AI algorithm encoded into silicon, are inert physical objects.
Every evaluation we are doing here is of a complex system that involves the interaction of people and machines and physical connections and so on.
An aviation system connected to every country and region with millions of people and machines involved is no less complex than what’s under discussion here and no more deterministic.
We don’t regulate airplanes because they aren’t people. We regulate pilots and mechanics and leaders of the companies that make and own them.
The task at hand is to regulate the people involved in AI to get the outcomes we want.
The idea that AI can’t possibly be addressed because it could autonomously break free and ruin something is fucking ridiculous.