Hacker News Reader

Comment by nicoburns | Hacker News Reader

IMO this is a fantastic demonstrattion of the potential dangers of AI. Of course a chat bot isn't that dangerous, but I can easily imagine future society putting an AI in control of things like the power grid or other industrial systems.

If we did this we'd probably put safegaurds in to make sure that the AI didn't do anything catastrophically stupid. What this very neatly demonstrates is that unless that safeguarding system is a completely separate non-AI based system that has power to override the AI then those safeguards will likely not be effective. It is no use trying to put safeguards within the learnt model.

View on HN

I’d say people are rapidly getting a feel on how these systems work and don’t work. Maybe we all agree soon that nothing serious should be controlled by this kind of AI (self-driving etc.)

Also we should have hardware restrictions to systems controlled by software to mitigate Therac-25 style accidents. Things can still go wrong but would be nice to have multiple levels of protection. Traditional software checking AI software, hardware checking traditional software.

I would love to give large language model control access to my spotify though. Especially if there is undo button. These systems now are great for toy use cases.

>Maybe we all agree soon that nothing serious should be controlled by this kind of AI (self-driving etc.)

I think there is essentially no chance of this happening, the cat is very much out of the bag.

It depends what you mean by "this kind of A.I."

Nobody is going to ask a chatbot trained on the text of the internet to drive a car. How would that even work? What mission critical A.I. do you see being trained by reddit conversations and dungeons and dragons manuals?

Social media moderation seems like a likely candidate for its actual use and also one where its failures could cause significant harm.

Um, sorry? How could you ever have a car driven via an LLM? I think the cat is very much staying in that bag.

surely they’re referring to using them as input for driving instructions

That doesn't even make sense.

it doesn’t make sense because you aren’t thinking about it correctly

> I’d say people are rapidly getting a feel on how these systems work and don’t work.

For the vast majority of people out of tech it's magic and they have absolutely no ideas of the limitations and implications of such tech. Even here on HN it's very obvious a lot of people don't understand even the basics of it

Well for me atleast it is magic. I understand roughly how it is supposed to work, but the output is too good based on that. Not in a million years does it feel ready for serious tasks though.

If I use reporters as proxy for layman. This has sparked countless news paper articles and majority of them I feel spend alot of time going through the caveats with the limits. Mostly the lack of logic and hallucinations.

My naive view is that people can judge the chat system using their interpersonal skills and actually get a feel of flakiness of the AI better than reading statistics on the subject.

I am optimistic people are becoming aware of the danger because we have seen a lot of recent and practical ways to remove protections in something like ChatGPT. I think a true AI will include the ability to change, like people do. The danger is other people changing you, radicalizing you.

An extension of this: Even if we avoid putting AI's in charge of critical infrastructure, I think it would be within the power of an AI to describe to someone how they would best attack an isolated system. It would certainly be possible for an AI to social engineer 1 or more targets, and to convince better-connected resources to act on its behalf.

Maybe we're about to see a renaissance in the studies of ethics and humanities, as it becomes ever more important to value life and right-from-wrong as we rush to AI armageddon. I promise you I'm not normally one of the doomsayers.

I think it's important to understand that not much resource has been allocated towards AI safety thus far.

Both in a positive sense, in terms of the amount of low-hanging fruit yet to be harvested, but also a negative sense, in terms of the % of R&D spend that has been allocated by unregulated/market-driven approaches.

Right now these systems are just toys, but soon they will have meaningful capability to do harm. I think we'll need some regulations to produce an appropriate level of safety before too long (concretely, I think we'll want this before the end of the decade, probably sooner).

For example, I think it's substantially more dangerous to attach a LLM to the internet; what's stopping OpenAssistant from taking on a script kiddie personna and trying to hack external systems?

For a simple LLM like ChatGPT we might not care, but I think full interpretability should be our gold standard here. We're nowhere close to understanding why a given utterance is made, nor what the "internal mental model" of a given character being simulated is. Could we detect a case where a LLM is simulating a deceptive personality that's secretly trying to access your AWS account so it can spin up more copies of itself? Until the answer is "yes, with five nines accuracy", I am nervous about the future of this technology.

"Studying the ethics and safety of ___ has not received as much resources as ___ itself" is hardly unusual.

We're in the midst of a mass-extinction due to habitat destruction, climate change, and pollution...all because progress and growth was more important than ethics, impact, or safety.

That's because common sense isn't common. In the past, various nations wanted to be king of the world. They still do, even if it's just king of the ashes.

Reply on news.ycombinator.com