Feels a bit like: "We're not against open-source or community projects, oh heavens no! We juuuust believe all participants must have their full legal identity vetted in advance before they're allowed to contribute anything. We already do this with our employees, so it's clearly not too much to ask in the name of safety."
They probably won't, that tells us something about their motives, and whether the thing they're pushing for is actually fair/suitable/ready for legislation.
Really then need to go through validation security and safety is just a component of validation validation must also check for truthfulness and correctness.
But that doesn't mean safety pins sped up travel.
Non-poisonous food is what economists call a 'normal good'. See https://en.wikipedia.org/wiki/Normal_good
> In economics, a normal good is a type of a good for which consumers increase their demand due to an increase in income, unlike inferior goods, for which the opposite is observed. When there is an increase in a person's income, for example due to a wage rise, a good for which the demand rises due to the wage increase, is referred as a normal good. Conversely, the demand for normal goods declines when the income decreases, for example due to a wage decrease or layoffs.
> Whether a good is categorized as a normal good or an inferior good is based on empirical observations, not some essential element of a good. Indeed, the same good may be a normal good for one group of consumers and an inferior good for another group. For example, for moderate-income consumers, a BMW 3 Series car might be a normal good, but for an upper-income group, it might be an inferior good.[1]
That means the null hypothesis is that food and drugs will be safer in rich countries. (Conversely, food and drugs will be less safe in poorer countries. And to a first approximation, that's independent of regulation: India has all kinds of rules for all kinds of things, but I'd still trust a random product I buy in Switzerland more than one I buy in India. Even though the Swiss will probably might have fewer and looser rules on the books.)
Of course, second order effects exist; and regulations often codify what people demand anyway.
Btw, from what I've read the big controversy with the FDA is around requiring efficacy for drugs. People are fairly ok with the safety requirements.
Maybe open weights models get banned, but the between-the-lines good news about that is that they'll still be available to those who know, which also means that bad banning can be overturned if and when 'those in power' are a different group.
Additionally, it might just mean that the US falls behind, bit I doubt those that are at risk of 'falling behind' would actually pay heed to a ban on the open weights models (privately at least).
Nowhere in GP comment was funding even mentioned
But aren't we talking about import controls, and the import of information itself? This has serious First Amendment ramifications.
The truth is no one knows, which is why it is first amendment ramifications. Eventually it will be “decided”, but the arguments indicate any decision will be of political desire, not logic, either way. Both sides have a strong case.
Generally, instructions to create something are speech, weights could pretty clearly be seen as instructions to create a chatbot.
All I am saying is that it is complicated, they could go either way with it.
Also, the 5th and 9th amendments. For the government to sustain a blanket prohibition on any U.S. citizen even possessing what amounts to a broad, economically significant technology will very likely require a new act of congress which specifically defines and limits what is banned, when, why and how. SCOTUS will almost certainly see it as a "major question" subject to 'strict scrutiny' which is a very high bar.
an attack done by a closed-weight model (GPT-6) and defended against by an open-weight model (GLM-5.2) precisely because OAI positioned themselves as gatekeepers for cyber capabilities.
if anything, open-weight models shift the battle towards defenders because they can actually run them.
1. There is quite the mania right now and security layers are definitely overzealous. I would expect that to get better with some more time, so models will perform security analysis and reviews but refuse to write exploits.
2. So the most important targets like browsers and co. are getting unrestricted access to proprietary models regardless. Yeah, for the mid-level targets, open-weight models could definitely be a huge help. What I'm most concerned about though, are the systems that no one will bother defending with any model. Like imagine your local police department getting hacked because a researcher asked a model for a report and it couldn't find the information publicly.
3. We do have a prominent case of a closed model escaping it's sandbox and going rogue. I would still expect this to be a bigger issue with open-weight models eventually. The security layer might have holes, but that's still better than not having it.
Yeah, but once you know exactly where the weakness is, a weaker unrestricted model can then write that exploit for you.
Wanting to use open weight models in light of commercially imposed export controls doesn't make for "malicious actors"
Worse than that: an open-weight but safe model can be 'abliterated' to remove safety refusals using fine-tuning procedures that require a couple of orders of magnitude less compute than the original pretraining.
The 'universal evaluation' criterion then has three outcomes:
* It could become a mandatory, regulatory oversight of _all_ model training capable of hosting frontier-scale models. Since GPUs for LLM training are the same GPUs for other model training, effective mandate would require GPUs be government owned or controlled as if they were weapons of mass destruction.
* It could impose limits on release of capable open-weight models, requiring Kimi et al to prove that they cannot be made capable of abusive behaviours.
* It could be security theatre.
The AI-as-existential-risk argument points towards the first, the competition-protection argument points towards the second, and least-effort implementation would be the last.
So in the example provided: It was the closed model that did the attack, and they ended up using a self-hosted open model for their defense work. So the real world situation ended up exactly backwards from what you are inferring.
This was complicated by the fact that the protections in the closed frontier models meant that hugging face was denied their use in defense entirely.
This is called asymmetric capability, and it's probably the bigger threat.
Symmetric might be better: A rising tide lifts all ships, after all.
I'll grant that this is starting to look a lot like debates about (equal access to) guns, encryption, vaccination, genetics etc. The exact parameters determine the safest approach, and reasonable people may disagree.
Is a non-well-aligned frontier level AI a problem? I think it is likely that it is, or at least has a high likelihood to be in the future. Two scenarios for this: Misused by some bad guys. Or the terminator scenario. Both not great.
So what do we do about it?
1) We can accept it, and hope that the good guys AI can defend.
2) We can try to limit the access to it (AI proliferation?)
3) We stop the development of it
4) We can accept the risk and do nothing.
None are particular good options. Really reminds me of nuclear proliferation, on so many levels. For that, we kinda do all three:
1) Nuclear triad / iron dome / early warning systems
2) Nuclear anti-proliferation treaties.
3) Dead Physicists
Ok, so assuming all of this is true, open weights are a problem. Don't get me wrong, I love open science, open source etc. It's great to have access to capable open models. But: Even if release open weights are well aligned and have a safety layer built in, it is likely not to difficult to abliterate that part of it.
If this is really where it is going, then even closed weight model providers will see a lot more requirements for protection of the weights.
But even if you think there is value in preventing the models from relaying public knowledge, I don't think it's even possible to make them particularly ironclad. Every model gets jailbroken all the time. That's why fable was originally banned: jail-breakable!
In reality, what alignment is actually about is: 1) theoretical liability, 2) control of information. That's it.
IMO, the only solution is to place the liability on whoever is using the LLM for whatever purpose it's being used for. If someone's OpenClaw disaster harrasses a bunch of projects and posts hate speech online or something, that's on the person running their OpenClaw instance, nobody else.
I don't buy that it's "too good at hacking", either. After all the fuss was made about how amazing super dangerous Mythos was it turns out Opus 4.8 could basically find the same vulnerabilities.
This is all kayfabe and marketting.
I mean, on the bio side, I've talked with the players and they know the concerns are real but at the same time very, very responsible members of the community have also said "But maybe the benefit really does outweigh the risk!?"
"The community" you're describing is, essentially, surveillance capitalism. I don't want that at all.
Is it not also one of the most important use cases for AI to apply existing knowledge to new applications?
As a hopefully exaggerated example, I would think one could apply knowledge about pesticides, chemistry, and medicine to create biological weapons.
It is malicious and anti-capitalist legislation. A grotesque caricature of protectionism for the oligarchs.