Hacker News Reader

Comment by dwa3592 | Hacker News Reader

I wonder if there would be an equivalent of Non proliferation treaty like Nukes?

I know it sounds crazy - but if there's even 0.1% chance that some models are so good that they can be used to hack into people's bank accounts - I, as the government, would not want that model to be publicly accessible. I would also request other countries to come to the table and sign this NPT(for AI).

Public will still have access to smaller models (like guns etc) up to Opus 4.8 etc but anything bigger than that is sooo good that it's dangerous. Nuclear also has benefits but the governments consider the worst when making policies rather than the best.

I am not touting Mythos as the god model but I wonder if the policy will move in this direction.

View on HN

Bank accounts are currently regularly ""hacked"" by phish and other types of fraud. Current models are capable of helping with that.

Then there's monstrously stupid stuff like https://www.visa.com/en-us/solutions/intelligent-commerce , where visa place an AI inside the security boundary, pre-hacked for anyone who can prompt injection it.

I think Dario's about to learn very quickly that making overly-hyped statements around safety to generate market interest is likely to result in regulatory reactions that don't just entrench his position as a market leader, but also severely curtail his business options.

You can't claim that your models are so good that they're basically weapons and then act shocked when the government starts imposing export restrictions on them exactly like they do for arms manufactures. Dario's recent blog post on "saving democracy" via regulatory capture and banning open weight models also mentioned export controls but just for chips, not what he's selling.

I'm sure many countries have learnt lessons from the NPT and would have the good sense to not agree to the same thing again. We've seen time and time again if you have nukes you can do what you want to the countries without them.

Agreed! I think it will have to be a better version of the nuke NPT. Something like every country's government keeps a Mythos like model and that's it. We stop there. Since it's a model; the most powerful country could create as many copies of the model as the countries ready to sign the treaty. But that's it. No more development after that. Somehow countries will have to come together to work this out. What's the solution otherwise? These companies won't stop at any freaking cost.

The AI threat is clearly a subset of the irresponsible billionaires threat, and like that threat we basically have to wait until the first sufficiently lethal incident before there can be real action taken.

  > I wonder if there would be an equivalent of Non proliferation treaty like Nukes?

It work for nukes because production and scale needed is such that only state actor can do it. Obviously model training is getting expotentially more expensive, but its still nowhere as hard.

On top of this is just gonna be very hard to steal a nuke. So country cant just steal a nuke if they cant build one.

And even if some country do steal one nuke what of it? It really gives them nothing because they wont have parity for MAD.

Stealing model weight doesnt sound as complex though - once weights are out any small company can abliterate and run as many instances as they want.

Onbviously its not super simple, but certainly doable no matter how much effort LLM companies put into securing weights.

Nukes are much cheaper to build actually. And easier. Most countries that have nukes probably couldn't train a frontier model, and that's considering you can "just buy" GPUs. Imagine if they had to make semiconductors too.

There is nothing that LLMs can do that humans cannot. If you are worried about bank accounts getting hacked, that's a problem to be solved on the banking side.

It's the scale problem (both speed and access) with LLMs. Don't you think the barrier to entry for hacking has become lower with LLMs?

Not really. Firstly, speed and access are actually money problems.

Secondly, motivated state actors have no dearth of money. It's not like the Chinese are saying "oh it would cost a bit too much to infiltrate all US financial systems, we'd have done it if it cost less."

If it can be done, it would've already been done. No party that matters is materially "unblocked" by the advent of LLMs.

I think I disagree with your premise.

If china as a state wanted to do something nefarious they would just do it. They have no interest in hacking regular folk's bank accounts. It's the criminal rings (around the world) who are interested in doing so and LLMs (and other types of AI models) can 100% help them scale their operations. I built a voice cloning app last summer with GPT4.5. It needed 5 seconds of sample to clone someone's voice and it was shocking that it was possible. I showed it to my parents and we decided to have a code word in case of emergencies. I told them not to believe if anybody called and asked for money pretending it was me.

You'd be surprised how many senior citizens fall for these traps every year.

if models are good at hacking software they are equally good at patching it

I'm pretty sure the days of any government signing any NPT with the US are very over. The trust is broken. I'd rather my government stockpile all the weapons of every sort at this point.

Reply on news.ycombinator.com