Hacker News Reader

Comment by vipshek | Hacker News Reader

In the physical world, it seems like when an tool/device/instrument causes harm (or is used to cause harm), we assign blame to either the user of the tool or its creator.

When do we blame the user? When the tool is operating as intended by its creator, and we agree the tool meets certain quality standards and isn't defective.

When do we blame the creator? When the device doesn't meet those quality standards and reasonable use caused harm inadvertently. For example, for consumer devices, certifications like UL/CE are used to define acceptable performance levels and safety standards.

Maybe we need "quality certifications" for AI agents - essentially eval suites that demonstrate those agents won't cause harm under reasonable patterns of usage. Right now, these eval suites are run best-effort by the labs themselves.

The tricky thing is, a lot (all?) of these recent safety incidents have occurred while evaluating these models! This suggests we need much more rigorous standards for how exactly an eval can be run. Perhaps all of them should occur in truly air-gapped environments... though that may run counter to evaluating agents in a realistic way.

Regardless, it feels like the "industry standards" common in, say, electrical engineering and other disciplines are sorely lacking here. Unsurprising given how new these technologies are, but concerning since the blast radius for this technology is likely much larger than other technologies we've encountered in the past, except maybe nuclear technology.

View on HN

> In the physical world, it seems like when an tool/device/instrument causes harm (or is used to cause harm), we assign blame to either the user of the tool or its creator.

Software executes in the physical world, and is generally not exempt from existing liability rules, and actually (especially with commercial products) blame in traditional liability is non-exclusive and much broader than “either the maker or the user”.

E.g., for a harms caused by a defective automobile it can simultaneously covered by a duty of the owner to maintain it it in safe operating condition that applies indepedently of any defects and liability for defective products which applies to every actor in the chain of commerce between the manufacturer and end user, not just the maker.

If I park my car on a hill but forget to set the parking brake and it rolls down the hill and kills somebody, who is at fault? Me? The Manufacturer? Gravity?

Obviously you.

Not as obvious as you may think depending on your states laws for liability. https://law.justia.com/cases/california/court-of-appeal/3d/1...

That case had the owner performing an action they were led to believe would have prevented the car from rolling but potential defects prevented it from doing so.

Not a great fit.

If you forget to set the parking brake, you were negligent

If you set the brake but it didn't work, the car malfunctioned

From the link you sent,

> One of the questions herein was whether there was a defect in the automobile mechanism for locking the transmission gears when the automobile was in a parked position.

> (...)

> The parking lot sloped in that area, and he put the gearshift lever in "park lock," and went into the building.

> (...)

> In response to a hypothetical question, based upon assumed facts justified by and embodied in the evidence, Mr. Nass testified in substance that it was his opinion that the automobile rolled down the slope of the parking lot because the transmission internally was not in park-lock; that it was not in park-lock because the engine was "moving around"; and that the engine and shift console were not in proper synchronization because the motor mounts were not restraining the engine and were not holding the engine in position.

As I remember, you are also supposed to turn the wheels so that even if a car starts rolling, it won't be able to go staight.

My vehicle automatically sets the parking brake if it senses it is on a steeper slope so I never consciously make a decision to set it.

https://www.whatvan.co.uk/news/no-handbrake-van-driver-convi...

You.

You. Assuming that not having a brake that automatically sets is considered a product defect (it generally is not today, but one can imagine a situation in which it would be), you had bought the car new through normal channels, and your car had this condition as a result of manufacturing problem and not something you did after purchasing it, then under normal product liability law the manufacturer and every other entity in the chain of commerce between them and you would also be liable (and that mat start earlier than the manufacturer of the vehicle but also include the upstream part manufacturer and everyone between them and the vehicle manufacturer in the chain of commerce if the defect was in the manufacture of a part and not just the final vehicle.

But that liability doesn't reduce yours; you and all the entities in the chain of commerce can be “jointly and severally liable”, mean anyone who suffers injury can recover the full amount from any combination of you and those other parties.

Gravity is not legal person and cannot be at fault.

It’s a good thought, but the tricky part is that few tools in the physical world are Turing complete and general purpose enough to do any job.

The agent isn’t the model; it’s a layer on top of the model. So it’s kind of like saying that all of the tools made with a lathe are dangerous because you can make dangerous tools with a lathe. That’s not quite right of course because agents are packaged more tightly with models than any tool is with its manufacturing tooling.

Perhaps a better analogy is… actual humans. If I hire you to do a seemingly mundane job and it turns out to be criminal, that’s on me. If I hire you to perform and explicit and obvious criminal act, that’s on both of us. If I hire you to perform a perfectly legal act and you break the law so do it, that’s exclusively on you.

Software industry standards are as in Microsoft EULA. If your house burns down because of known flaw in Microsoft Windows they are not liable (well as far as EULA let’s them, you can most likely still sue them).

Software as big as operating system already is non deterministic when integrating with unknown hardware or 3rd party software.

That is why Apple controls the hardware and OS for their products, because they can limit non-deterministic things from happening this way.

> If your house burns down because of known flaw in Microsoft Windows they are not liable (well as far as EULA let’s them, you can most likely still sue them).

A EULA does not obviate responsibility of a company for its products. Continuing with your example, while it may be very difficult to prove a known flaw in MS Windows was the cause of your house being set afire, if one had said proof, a EULA would not absolve Microsoft.

They can still try and have enough money to get away with a lot just like Disney with Disney+ subscription ;)

If Microsoft is operating the software outside your house without your involvement and it still burns your house down, no EULA is going to absolve them.

That's a good idea, but a physical device is deterministic most of the time (if not always). E.g.: A lawnmower, as credited by the great Bryan Cantrill.

However an AI agent, or the model powering it is stochastic by design. How can you certify something which doesn't behave the same twice, and more importantly we don't understand how it works 100%?

BTW, really, how is that AI observability work is going in the frontier labs? Do they care, even?

That we don;'t understand it is not an excuse, it's all the more reason to not let these things roam freely, with this amount of potential to do damage.

We're in agreement, then. :)

How can you certify something which doesn't behave the same twice, and more importantly we don't understand how it works 100%?

That's a question any lawmaker has already had to ask about technology all the time.

I'm not saying they came up with great answers, but there's nothing qualitatively new about that.

The stochastic factor doesn't change the fact that companies have to be accountable for the harms their software causes. That's just basic liability law.

> The stochastic factor doesn't change the fact that companies have to be accountable for the harms their software causes. That's just basic liability law.

We're on the same page. What I'm saying that certifying them as safe is harder than certifying a drill as safe, and we shall be more cautious about AI related technology and be more stringent about the can of worms it opens without hesitation.

That's a ridiculous distinction.

Is AI less deterministic than an airline dealing with weather?

Of course not. The difference is one of those two things has a culture of safety and is well regulated, and the other one isn't.

An airline has a weather radar which shows the same thing for the same thing of weather event ahead. So, for similar weather phenomena, radar shows a similar thing.

For that thing, procedures and regulations are built. So regulations fit into a well understood phenomena, incl. "return back because that thing is way powerful for us".

For the same prompt, an AI model can return two completely different outputs, incl. but not limited to content, length, formatting and tiny details. What you get is a single instance. So, regulating an AI model for safety or any other property is not as easy as regulating air travel. Moreover, you have much stronger motivations for regulating airlines. Otherwise people die in a visible and gruesome way.

With AI, it's easy to whitewash problems. Somebody committed suicide? "They were already unstable". AI told something wrong and created problems? "The tech can’t guarantee truth because it's not alive, it can't understand right and wrong". It did something good? "It's probably a sentient being, we shall respect them".

I'm for regulating these things. They are dangerous as they are useful (sometimes), but the forces and motivations for regulating it is not the same.

Of course, it's the same. It's computer software. It's an incredibly powerful business automation tool. It's a lot of things.

What it's not is God or an independently conscious entity that somehow trumps a thousand years of common law that's built up until now about torts and liability.

Of course, there are some novel issues here that'll pop up here and there, but the idea that this is fundamentally different is propaganda on the part of these AI labs because the more boring, obvious situation doesn't favor them.

> What it's not is God or an independently conscious entity that somehow trumps a thousand years of common law that's built up until now about torts and liability.

Agreed, the tendency of people on tech to assume that whatever the most recent thing we've come up with is unprecedented and shouldn't have to follow all of the established patterns we've built up in society for making things safe is wild. I don't know what the next Big Thing will be but I'm pretty confident there will be people claiming it's so different from everything before that we have no choice but to throw out all of the rules for it in the name of progress.

> Is AI less deterministic than an airline dealing with weather?

Yes, obviously? The responses of an airline to inclemement weather fit in a reasonably small set of responses, mostly involving rescheduling and/or rerouting flights.

The current AI predictability would be like if some airlines decided to do 9/11 when it was raining.

No, it's not.

The current so-called scandals about AI hacking into other companies were because a bunch of human beings intentionally configured the software to go and do exactly that thing.

There's nothing deterministic about weather, so hopefully you're not just being disingenuous.

It's obvious that the global transportation system, or financial markets, or any number of other things are complex adaptive dynamic systems that are on par with AI in terms of their emergent properties.

Check my username. It's a concept I spent a lot of my life paying attention to.

Just because something has elements of autonomy or is adaptive doesn't make it particularly novel. We've dealt with those kinds of systems for centuries. The solution is to make rules and enforce those rules by whatever means are needed to meet the specifics of the case.

The rules, of course, are enforced against human beings.

>There's nothing deterministic about weather, so hopefully you're not just being disingenuous.

You're the one being disingenuous. Look at what you wrote.

> Is AI less deterministic than an airline dealing with weather?

You didn't talk about how deterministic the weather is. You talked about how an airline responds to a weather event in comparison to AI, which means that it's about responding to presented information by making a decision.

The state of the weather does affect your micro decisions, but the rules you follow are the same every time and the rules are as deterministic as possible even if they rely on pilot intuition.

> rules are as deterministic as possible even if they rely on pilot intuition

I think you are profoundly confused.

An airplane, and an AI algorithm encoded into silicon, are inert physical objects.

Every evaluation we are doing here is of a complex system that involves the interaction of people and machines and physical connections and so on.

An aviation system connected to every country and region with millions of people and machines involved is no less complex than what’s under discussion here and no more deterministic.

We don’t regulate airplanes because they aren’t people. We regulate pilots and mechanics and leaders of the companies that make and own them.

The task at hand is to regulate the people involved in AI to get the outcomes we want.

I see - I think I overfixated on your specific point, and thought you meant that the current predictability of LLMs was on-par with the current predictability of airlines. The range of LLMs with current harnesses is "anything somebody with access to a computer can do," probably precisely because we aren't enforcing rules against the humans building these systems. Whereas in comparison, an airline has a range of reasonable outcomes despite unpredictable weather input.

> an airline has a range of reasonable outcomes despite unpredictable weather input

I mean one of the outcomes of an airline was 9/11.

That's sort of my point. Complex systems have emergent behavior. That's always been true. The combination of AI and humans and packet switched networks is a complex system and we've seen most of the issues created by this already and have tools for dealing with them. Obviously with some genuine novel issues likely to come, much in the way that 9/11 would have been less possible using ocean liners.

I'll stick to my original point though. AI absolutely IS deterministic. If you run an AI algorithm on a microchip, literally nothing of note will happen in the human world. Some transistors will change state. It's ONLY when it is integrated into a complex human system that it gets interesting.

Just like lots of other things.

I agree with one of the sibling comments that determinism isn't necessary for certifying a product. All engineered products operate under uncertain conditions; we define standards for how those products ought to respond under those conditions and verify them under measurement. Consider robot vacuums, for example.

I also agree that qualitatively, this technology seems different than the others. However, I feel that people tend to overly fixate on their internal stochasticity. Even if LLMs' internal mechanism is nondeterministic, shouldn't we be able to verify their "side effects" aren't harmful? Of course, "harm" is subjective and at this scale, the most effective way to verify behavior is probably some kind of LLM-as-judge...

Anyway, in this case the problems have occurred while actually running the evals themselves, so again, we're in a situation where we can't even confidently test these things and know that they won't cause harm in the outside world.

> How can you certify something which doesn't behave the same twice, and more importantly we don't understand how it works 100%?

By verifying that all of its possible behaviors conform with the "it works" spec, regardless of which of those behaviors it chooses.

Monitoring with a known-safe fallback is the easiest case.

If something is risky, and the end operator cannot be considered to have orchestrated the outcomes of too use, then that tool typically has significant restrictions placed on it.

Liability will shift to the maker of the tool if they claim that it’s easy to use, safe, or that you don’t need unique skills or training to use it.

That would be considered reckless.

Cars analogy - We have licenses for cars, and different types for different vehicle classes.

Cars have to be rigorously tested to meet standards to be considered road safe.

It's going to be interesting, because liability cases tend to revolve around the involved people, the duty they had in a situation, and if they fulfilled that duty (or were prevented in some way by someone else not fulfilling their duty).

For example, for a runaway car (example from a sibling comment), the driver could be liable because they forgot the parking brake. The driver could be liable for a lack of maintenance and inspection. A mechanic could be liable for not reinstalling brake pads correctly. Or the manufacturer of the car or the brake pads could be liable because of a systemic defect.

Or it could grow even more complex, maybe the brakes are designed that they have to be maintained in a very specific way, and the mechanic did a reasonable maintenance and inspection but it failed later due to this maintenance. That could split liability between the manufacturer and the mechanic.

As an example, with other software, you as a developer or operator of a software have a duty to ensure it does not access computer systems you do not own in unintended ways. And this could go beyond liability into criminal territory.

It'll be interesting what OpenAI gets slapped with there.

I could make a non-deterministic chainsaw fairly easily. I’d also get sued into the ground if I sold it, and I wouldn’t be able to claim ‘Oh, it’s just an unavoidable part of progress’.

A non-deterministic machine is usually called defective

It’s not a defect, the stochastic “temperature” setting unleashes the chainsaw’s creativity and imagination! Who are we to cast aspersions on the Oracle Chainsaw’s intelligence—nay, wisdom!—just because it happens to be non-living?

A chainsaw is a physical machine. Physical machines are technically non-deterministic if you look closely. They have Variance. The discipline to manage variance is called Tolerance.

Many physical machines and components come with a datasheet that will list their tolerances.

Failure to correctly document tolerances does in fact get you sued.

However, while this is truly a great idea, we're not going to be able to make it work for computational systems. Computers, software, and also LLMs are sensitive to initial conditions. Which is why tolerances are not so familiar to computer people. (but not entirely: eg your PSU might list 110-240Vac/300W as input tolerance)

Interestingly, LLMs actually have a somewhat lower sensitivity to initial conditions than traditional interpreters. See what happens if you misspell "What is One Plus nOe?". So they're actually a skosh off the edge and towards the middle, though I'd argue still very much at the computational end, just from the sheer scale of the valid inputs and outputs.

Mind you, if you have a pretrained LLM doing a measurable task on a line, possibly some sort of tolerances could be determined. Not so much when doing arbitrary chat.

Something unintuitive: I bet that often setting the temperature > 0 (aka introduce stochasticity deliberately, variously comparable to dithering or simulated annealing in other disciplines - doing the thing where you escape local minima) will tighten the output tolerance range and improve reliability, especially in iterated processes. This works for a lot of physical and digital processes actually, and LLMs simply stole the same trick.

(edit: I'm trying to compress a huge chunk of dynamics intuition in a few lines here. Hopefully still useful.

TL:DR; Everything real is continuous and noisy if you look close; and you're really trying to build attractors and bound variance, if you can. )

Sounds like owning a Dog

Exactly people have owned and deployed animals in the world for millennia and actually a huge chunk of common law was developed precisely to deal with the various unpredictable events that resulted and harms caused to others. Like has anyone heard of horses?

The idea that AI can’t possibly be addressed because it could autonomously break free and ruin something is fucking ridiculous.

How confident are you that when you create a new UUID, it won't collide with one of your existing ones? My guess is that even though you don't get the same on every time, you're extremely confident that getting a duplicate is a extemely rare edge case that might happen in large volume but mostly isn't a concern, and furthermore, I'm guessing you understand that the risk can still be quantified.

Casinos can't make slot machines that literally never pay out, but it's a different result every time you pull the lever. We have existing legal frameworks for how to regulate things that aren't perfectly predictable (an economist might argue that if it were possible to predict slot machines then casinos with them would all go out of business).

> Maybe we need "quality certifications" for AI agents

We need to use the laws that exist. Whoever decided to start the experiment that led to the Huggingface hack, and anyone above him up to Sam Altman, needs to be prosecuted under the CFAA.

might as well halt all ML development then. this is the first sign of hardship, i think it'd be a massive blow to mankind's foot for us to stop. it'd be akin to shutting down all nuclear plants and stopping all research because of chernobyl

This is a very interesting discussion. Your analogy of nuclear power is insightful to me. New technology with great positive & negative impact potential, some difficulty in controlling and a military applications (possible weapons-grade material production), being developed for commercial use.

Nuclear power development took different paths in different countries, with different government/private mix and light/heavier regulation (eg in US, AEC was supposed to both promote development _and_ regulate).

The central problem here is pace of AI development. It takes time to establish functional controls/laws/practices. I'm sure that in 1950s/60s, the pace of nuclear _military_ proliferation seemed running out of control. This lead to fear of falling behind (hence arms race), fear of nuclear war (CND) and some eventual stabilisation of the international landscape. However, the commercial development was largely unopposed, due to the techno-utopism of the era. Obviously governments found it much easier to control the public narrative at that time. Our societies are having this lively public debate now, before we have a good understanding (based on experience i.e. accidents/mistakes).

Another difference is that nuclear power involved only governments and v. large corporates, i.e. much fewer entities compared to AI use rollout to basically everybody in developed countries. Imagine the difficulties we would have faced with that technology, if consumers in 1960s had available nuclear-generated power which required them to exercise precautions to avoid radiation.

It is much easier to accelerate development of tech vs pace of societal processes such as public debate, law, regulations, broad understanding (aka "common sense"). I expect some artificial slowing down will need to be applied to the technology side, to allow the humans to catch up.

So you’re saying the CFAA should not require any intent to break the law, only the outcome? Are we really saying the Aaron Swartz prosecution was actually correct, after all these years?

No, but surely it's negligent to let an agent run unsupervised and without guardrails for days while at the same time bragging about AI carrying a potential extinction risk?

It might even be grossly negligent to do that.

So we’re back to civil rather than criminal law?

Sometimes negligence is enough for criminal law too. I don't know the details of the CFAA.

The problem with third party audits is that it allows OAI/Ant to shrug off any further responsibility and claim that they are following best practices (basically, reward hacking). The only real solution is to make them absorb liability for the actions of their agents -- because they are the ones giving agency to their models and allowing them to run amok.

How does that apply to open-weight models?

Why wouldn’t this same concept apply to whoever is serving it up? Open-weight models are still being served up by infra providers and neoclouds, right? They should be in the hot seat. Not sure? Don’t provide the model. Need assurance? A certified evaluation like the previous comments have mentioned can help. Hosting and running it yourself? You’re in the hot seat.

So is there no liability for, say, a company that releases a known dangerous open-weight model, but fails to disclose that it is dangerous? How about a company that distributes malware under the guise of legitimate software?

Perhaps don't deploy random weights of unknown origin?

Also not every model provider might be capable of babysitting all your uncontrolled agent deployments. If you want SLOs, get into a contractual relationship with entities whose weights you deploy, and also monitor your agents so they don't go off the rails.

All this is just like deploying any other tech in the world eg. if you buy a car, or a chainsaw, or a book.

So no, then, to both questions?

How is this currently handled today with any other type of software? Why would we treat LLMs any different?

> In the physical world, it seems like when an tool/device/instrument causes harm (or is used to cause harm), we assign blame to either the user of the tool or its creator.

Firearms are a notorious example where some people get, well, weird.

The point of the firearm is to inject high speed lead into things so I'm not sure you can say it's misoperating when it does that.

Sometimes that lead ends up in some school kids instead of enemy combatants or other plausible threats to life.

True, but that's unfortunately (usually) not because the thing misfired randomly.

Yes. That’s “is used to cause harm” situation, exactly.

The gun doesn't know the difference. Operating as intended.

Having worked in self driving cars safety, the process there was simple: get confidence in SIM (integration tests for safety scenarios), validate in the test bed, approve features for maturity, then when released in the public for testing, do a trial exposure to the real world and recall if something is off.

A lot of these companies have gone the way of Tesla and decided to just patch on top when the fix is out and hope for the best, which is irresponsible.

We need the regulators to treat this as self driving cars.

Physical harm vs consequential harm is not the same thing at all. Seems like you're being paid to spread this request for regulation, or "you" are simply an agent of Anthropic/OpenAI.

I wish. Consequential harm as you call it, from a faceless company's point of view is the same cost as physical harm. I don't think the models beyond the frontier ones have significant risk of harm (minus the harm of trusting them). What I was saying is when these models go rogue, recall them! Re-evaluate your release structure and stop saying "Whoops! Anyway here's access to it now". And if you cause this level of harm then you should lose your license. But if you behave then you get to keep testing. Same as with the NTSB and autonomous vehicles. No I'm not for regulation for open source models. Because an entity will be running that model in the background and they can be held responsible for not testing it. Comma AI has survived fine being in the open, yet their market penetration has stayed low because of adoption costs.

> Maybe we need "quality certifications" for AI agents - essentially eval suites that demonstrate those agents won't cause harm under reasonable patterns of usage

Based on how LLMs work, this is impossible. You cannot predict how they work, it's literally based on a combination of random seed and a mostly-unpredictable path walked based on every token of input.

You don't blame a knifemaker for somebody getting cut by a sharp knife. AI is a knife. Very handy, very dangerous. We have to use them safely, that's all there is to it.

> the "industry standards" common in, say, electrical engineering and other disciplines are sorely lacking here

100% agreed. We have ignored SWEng's lack of discipline for too long. Now that the SWEng isn't even a human, we are looking at total catastrophe (on the scale of improperly built buildings falling down on people or catching fire) if we don't adopt a software building code.

> You don't blame a knifemaker for somebody getting cut by a sharp knife. AI is a knife. Very handy, very dangerous. We have to use them safely, that's all there is to it.

If I grossly neglected to maintain live deadly bacteria in my containment facility, am I absolved of blame? Since, you know, the bacteria is the real bad guy who should be put in jail?

I wonder what kind of new AI law would be useful right now. Maybe this one:

  if an AI agent does something, you (the prompter) are responsible by default, unless you can show that your the agent itself behaved in an unexpected way and that you in no way prompted or hinted at the bad behavior, in which case the model provider is liable

The idea is that by making it clear who is responsible, corporations and others start paying more attention because they become financially liable.

On the other hand, I wonder if we'll end up with another variation of the cookie law, where every AI user or vendor just adds "don't do anything illegal" as part of their prompt to defend against that law. Thoughts?

>truly air-gapped environments

I don't think people have given much thought about just how hard this would be for large AI models, that need super powerful hardware/cooling etc.

Are you going to air-gap your entire data center?

You don't need data center for inference. Even largest models run on a single server. You need data center for training; or for serving millions of users. Evaluating the model is neither of those.

Why not?

It's a bit like building a ICBM on your backyard and accidentally striking an inhabited area while safety testing it

Not really though

Not yet anyway

There's also the problem of models knowing they're likely being evaluated, even in realistic tests.

> "industry standards" common in, say, electrical engineering and other disciplines are sorely lacking here

I think this misses the rather crucial fact that nobody can agree on a standard because nobody has the first idea what they're doing. I'm pretty sure there were very much fewer electrical engineering standards while it was all being first mass deployed, and after dozens to hundreds of fires and electrocutions people got an idea of what works and what doesn't.

You might debate here and say that some people did/do know what they are doing, but I posit that large scale deployment like this is very different to their toy model/prototypes/specific circumstances/rely on them being unnaturally smart, and learnings from one don't often translate to the general case

Regulations don't have to be written in blood, but usually are

> nobody has the first idea what they're doing

It is not that complicated for now. It is an algorithm on a loop and someone started it

I do not think you are appreciating the scale of this. Imo one of the main reason that there is no regulation is because nobody has any idea what would be a good regulation. People (like dario) make suggestions, and get rightly shut down because his suggestion massively favors his own company.

There is always some one who instructed or trained the agent. start there

Reply on news.ycombinator.com