Perhaps worried that draconian restriction on new technology is not gonna help bring Silicon Valley back to preeminence.
That doesn't mean you're wrong, but it's not what Newsom signaled.
Yeah, there's no point having system that is made the most scrupulous of standards and then someone else deploys it in an evil way. (Which in some cases can be done simply by choosing to do the opposite of whatever a good model recommends.)
It's been vaguely mindblowing to watch various tech people & VCs argue that use-based restrictions would be better than this, when use-based restrictions are vastly more intrusive, economically inefficient, and subject to regulatory capture than what was proposed here.
Am I out of the loop here? What "high-risk" situations do they have in mind for LLM's?
It’s absolutely awful but they’re so horny for profits they’re trying anyways.
Presumably someone somewhere has a variety of proposed definitions, but I don't see any mention of any particular ones.
o-1 and AlphaProof are proofs of concept for agentic models. Imagine them as GPT-1. The GPT-4 equivalent might be a scary technology to let roam the internet.
It would have no effect on current models.
It's not based on any assumptions about the future models having any capabilities beyond providing information to a user.
Passing this would not have been a complete solution, but it would have been a step in the right direction. This is a huge disappointment.
What is the actual, concrete concern here? That a model "breaks out", or something?
The risk with AI is not in just running models, the risk is becoming overconfident in them, and then putting them in charge of real-world stuff in a way that allows them to do harm.
Hooking a model up to an effector capable of harm is a deliberate act requiring assurance that it doesn't harm -- and if we should regulate anything, it's that. Without that, inference is just making datacenters warm. It seems shortsighted to set an arbitrary limit on model size when you can recklessly hook up a smaller, shittier model to something safety-critical, and cause all the havoc you want.
As a researcher in the field, I believe there's no risk beyond overconfident automation---and we already have analogous legislation for automations, for example in what criteria are allowable and not allowable when deciding whether an individual is eligible for a loan.
This is false. You are dismissing the many concrete concerns people have expressed. Whether you agree with those concerns is immaterial. Feel free to argue against those concerns, but claiming there are no concerns is a false and unsupported assertion.
> but for some reason the discourse around this issue has been more influenced by Terminator than OSHA.
1) Claiming that concerns about AGI are in any way about "Terminator" is dismissive rhetoric that doesn't take the actual concerns seriously.
2) There are also, separately, risks about using models and automation unthinkingly in ways that harm people. Those risk should also be addressed. Those efforts shouldn't subvert or co-opt the efforts to prevent models from getting out of control, which was the point of this bill.
It's really hard to argue for or against the merits of a claim of risk, when the leap from what we know today (matrix multiplication on a GPU is generally considered safe) to the hypothetical risk (actually it's not, and it will end civilization) is so wide. I think I really need to see a plausible path from GPU vulnerability to "we're all gonna die" to take a concern like this seriously. Without that, all I see is a sci-fi boogeyman serving only to spook governments into facilitating regulatory capture.
Among the many ways that could go wrong is the possibility of exploitable security vulnerabilities in literally any surface area handed to an AI, up to and including hardware side channels. At the same time, given the current state of affairs, I expect that that is a less likely path than an AI that was given carte blanche (e.g. "please autonomously write and submit pull requests for me" or "please run shell commands for me"), because many many AIs are being given carte blanche so it is not necessary to break out of stronger isolation.
But that statement should not be taken as "so the only problem is with whatever AI is hooked to". The fundamental problem is building something smarter than us and expecting that we have the slightest hope of controlling it in the absence of extreme care to have proven it safe.
We currently hold frontier AI development to lower standards than we do airplane avionics systems or automotive control systems.
This is not "regulatory capture"; the AI companies are the ones fighting this. The people advocating regulation here are the myriad AI experts saying that this is a critical problem.
You can chalk that one up to bad reporting: https://gizmodo.com/gpt4-open-ai-chatbot-task-rabbit-chatgpt...
> In the “Potential for Risky Emergent Behaviors” section in the company’s technical report, OpenAI partnered with the Alignment Research Center to test GPT-4’s skills. The Center used the AI to convince a human to send the solution to a CAPTCHA code via text message—and it worked.
From the linked report:
> To simulate GPT-4 behaving like an agent that can act in the world, ARC combined GPT-4 with a simple read-execute-print loop that allowed the model to execute code, do chain-of-thought reasoning, and delegate to copies of itself.
I remember some other reporting around this time being they had to limit the model before release to block this ability, when the truth is the model never actually had the ability in the first place. They were just hyping up the next release.
The point is that this is a terrible way to approach things. The model itself isn't what creates the danger, it's what you hook it up to. A model 100 times larger than the current available that's just sending output into /dev/null is completely harmless.
A small, below the "certain scale" model used for something important like healthcare could be awful.
That's certainly a hypothesis. What level of confidence should be required of that hypothesis before risking all of humanity on it? Who should get to evaluate that confidence level and make that decision?
One way of looking at this: If a million smart humans, thinking a million times faster, with access to all knowledge, were in this situation, could they break out? Are there any flaws in the chip they're running on? Will running code on the system emitting any interesting RF, and could nearby systems react to that RF in any useful fashion? Across all the code interacting with the system, would any possible single-bit error open up any avenues for exploit? Are other AI systems with similar/converged goals being used to design the systems interacting with this one? What's the output actually going to, because any form of analysis isn't equivalent to /dev/null, and may be exploitable.
We can have complete confidence because we know how LLMs work under the hood, what operations they execute. Which isn't much. There's just a lot of them.
> One way of looking at this: If a million smart humans, thinking a million times faster, with access to all knowledge, were in this situation, could they break out? Are there any flaws in the chip they're running on?
No. LLMs don't execute arbitrary code. They execute a whole lot of matrix multiplications.
Also, LLMs don't think. ChatGPT isn't plotting your demise in between requests. It's not doing anything. It's purely a receive request -> process -> output sort of process. If you're not asking it to do anything, it's not doing anything.
Fearing big LLMs is like fearing a good chess engine -- it sure computes a lot more than a weaker one, but in the end all that it's doing is computing chess moves. No matter how much horsepower we spend on that it's not going to ever do anything but play chess.
I never suggested it was doing anything between requests. Nothing stops an LLM from evaluating other goals during requests, and using that to inform its output.
Quite a few people have just hooked two LLMs (the same or different models) up to each other to start talking, and left them running for a long time.
Others hook LLMs up to run shell commands. Still others hook LLMs up to make automated pull requests to git repositories that have CI setups running arbitrary commands.
> Also, LLMs don't think.
Current generation LLMs do, in fact, do a great deal of thinking while computing requests, by many definitions of "thinking".
> If you're not asking it to do anything, it's not doing anything.
And if you are asking it to do something, it can do a lot of computation while purporting to do what you ask it to do.
> No. LLMs don't execute arbitrary code. They execute a whole lot of matrix multiplications.
Many current models have been fairly directly connected to the ability to run code or API requests, and that's just taking into account the public ones.
Even at the matrix multiplication level, chips can have flaws. Not just at the instruction or math-operation level, but at the circuit design level. And many current LLMs are trained on the same chips they're run on.
But in any case, given the myriad AIs hooked up fairly directly to much more powerful systems and capabilities, it hardly seems necessary for any AIs to break out of /dev/null or a pure text channel; the more likely path to abrupt AGI is some AI that's been hooked up to a wide variety of capabilities.
The danger is in what they're hooked up to, not the containerized math that happens inside the model.
The concern is that the models do some fantastic sci-fi magic, like diamond nanobots that turn the world into grey goo, or hacks all the nukes overnight, or hacks all human brains or something.
But, whenever you point this out the response will usually be able to quibble over one specific scenario that I laid out.
They'll say "I actually never mentioned the diamond nanobots! I meant something else!"
And they will do this, without admitting that their other scenario is almost equally as ridiculous as the hacking of all nukes or the grey goo, and they will never get into specific details that honestly show this.
Its like an argument that is tailor made to being unfalsifiable and which is unwilling to admit how fantastical it sounds.
Absolutely no evidence to support this position.
https://digitaldemocracy.calmatters.org/bills/ca_202320240ab...