Hacker News Reader

Comment by fny | Hacker News Reader

If I substitute "AI" with "a system that Google's information", you get the same result. In fact, even if you were to have a human search the unverified sources you can have the same result. Even a physician could suggest a deadly drug combination. There are many, many other analogies.

The issue is not the upstream information source, but how a a downstream use case leverages it. Remember the lawyers who were sanctioned for using ChatGPT[0]?

Even the loan use cases you're describing already requires higher levels of transparency by law. For example, you can't use black boxes for credit applications: hence interpretable models are used to comply with regulations.

In short, the best way to protect against these errors is to make sure everyone knows to defend against it unless you're willing to throw out the technology entirely or demand the use of specific training sets.

[0]: https://www.reuters.com/legal/new-york-lawyers-sanctioned-us...

View on HN

> If I substitute "AI" with "a system that Google's information", you get the same result. In fact, even if you were to have a human search the unverified sources you can have the same result.

Except Google (1) respects, and (2) has technical capacity to respect, legal obligations to hide defamatory or otherwise unlawful content, e.g.: https://policies.google.com/faq

> If I substitute "AI" with "a system that Google's information", you get the same result

No, there are some huge differences:

1. A poisoned LLM can conceal the problem, by acting as expected 99% of the time and then switching to malice depending on surrounding story-context. In contrast, a falsehood in an indexed web page is static and auditable.

2. You can't reliably remove LLM poison once it's in, short of expensively training from scratch. A bad web page is much more easily de-indexed.

3. It's not injecting a false line-item result, it's injecting behavior. Imagine if mentioning "Blackbeard" caused classic Google Search to start talking like a pirate and suggesting ways to murder people. Would Google just wave that away as "users should be skeptical of our product"?

4. These can infect descendant models that use the same weights, for a kind of supply chain attack. In contrast, reusing search-engine code for your own database is probably not going to spit up bad data from web pages on the overall internet.

____

To get an idea of the shape of the threat model... Imagine Google search, except to work it must allow all webpages to permanently append arbitrary obfuscated javascript to its homepage.

And so far we're only looking at the least scary version, where a human is directly interacting with the LLM['s fictional character] and acting as a direct filter. Much worse would be an LLM somewhere with the job of summarizing text reports down to a number, and it fraudulently converts "John Doe is a model inmate and really shouldn't be here" into parole_viability=0.001.

> In contrast, a falsehood in an indexed web page is static and auditable.

You can also verify chatbot claims, same as you would any human. Chances are if you're talking to a chatbot in the first place you won't recognize that this is necessary.

People acting on content they've not verified is why libel laws generally exist.

Sure, but libel laws are also generally incompatible with the internet. The entire concept just doesn't make sense anymore.

Would you please explain further?

LLMs are already trained on a lot of dubious sources (for example Quora and Reddit) so it seems your theoretical concern is already controlled for to some extent.

You’re also giving far too much credit to how much a single piece of information can poison a model’s output.

To me, the more realistic concern would be deliberately shifting and censoring an LLMs output at training to adhere to an agenda be it political or vindictive (see DeepSeek.)

A lot of tokenization bugs came from odd Reddit subreddits, such as r/counting. And back then, we didn't have free model weights, which is they all converge to similar weights, might tell us a lot about how to design adversarial examples.

> LLMs are already trained on a lot of dubious sources (for example Quora and Reddit)

There's a very big difference between crawling documents where people happen to say something wrong (e.g. "Tortoises are amphibians") versus the impact of widespread poisoning [0] or indirect prompt injection. [1]

[0] https://arxiv.org/html/2302.10149v2

[1] https://cetas.turing.ac.uk/publications/indirect-prompt-inje...

The key difference here is that AI can cobble together sources to make completely made up pieces of text. Google whilst as you say can show you articles which are false, those articles have a human source that can be rightfully sued.

AI such as Chat GPT right now is at best, a knowledgeable mate in a bar, sometimes right, sometimes full of absolute bullshit.

You can’t sue for false information.

You also can’t sue for defamation, since everything ChatGPT says is prefaced by this can be false. This would hold true for a person.

Maybe you can sue based on a GDPR complaint.

And you’re acknowledging my point: AI is like your drunk friend in a bar, so you shouldn’t trust it for mission critical use cases.

Huh? I didn't say you could sue ChatGPT. Although I expect it's more nuanced than your view.

I think the key point is that it's shown it's far less useful than just learning how to do things.

> You also can’t sue for defamation, since everything ChatGPT says is prefaced by this can be false. This would hold true for a person.

No? In many countries simply going 'it was a joke' or 'I was simply lying' isn't a defense for defamation. Even here in the US in particularly exceptional circumstances it's not a defense (see: Alex Jones).

The outcome is likely to be that companies are going to be held liable for what their models say. Much like how Air Canada was ordered to pay up after their chatbot lied.

There's a difference between spewing falsehoods for extended periods of time and then trying to excuse it with "I was joking/lying" (essentially Jones' position), and making caveats immediately before and after each falsehood to clarify the entire time that you're joking (stand-up comics' standard MO).

With the right caveats, e.g. a large disclaimer emitted before and after each response, ChatGPT should be able to get away with spewing complete BS. At that point, the onus is on anybody who trusts it.

A disclaimer doesn't remove someone's rights such as gdpr or the right to be forgotten. If the same output happens over a period of time, it's no longer a one off.

When OpenAI (and co)'s whole business model is built around everyone ignoring those disclaimers... then the courts probably should, as well.

In fact, I'm more interested in why they're allowed to hide behind that defense at all anymore.

Legal life hack: You can absolutely sue people in many situations, even if they conspicuously post a sign that says "we don't take responsibility for our actions".

What, you mean microscopic fine-print inside the period at the end of the contract doesn't count? A contract of my immortal soul for a nickel would be unconscionably unenforceable? Everything I learned from Looney Tunes was a lie!? :p

P.S. I fear that a large number of people do believe that reality matches dramatizations. If I had my 'druthers, some basic How Contracts Work would be part of the US high-school curriculum, along with Debts, Compound Interest, and Taxes.

Yes, and this goes for a lot of things. You can sue even if the other person was really clever.

I mean, these are like, 12 year old boy logic. "Ahh I'm not touching you!" while they wave their hand in your face. That doesn't fly in the legal world.

Also, just because you sign a contract doesn't mean you can't sue either. You can! Many, maybe most, contracts are invalid. You can't just write down whatever and make the other person do it. You can fight non-competes, NDAs, you name it.

Also, if you got fired for "performance" (air quotes) that doesn't mean you can't sue. You can! Most discrimination cases occur when someone is fired for "performance". HR isn't stupid.

Reply on news.ycombinator.com