Banning open weight models would be a disaster
rbren.substack.com
rbren.substack.com
Just like code was the battleground back then, open model weights are the new frontier. Think about it—code is just a bunch of instructions, right? Well, model weights are pretty much the same; they're the brains behind AI, telling it how to think and learn. Saying "nah, you can't share those" is like trying to put a genie back in its bottle after it's shown you it can grant wishes.
The whole deal with PGP was about privacy, sending messages without worrying about prying eyes. Fast forward, and model weights are about sharing knowledge, making AI smarter and more accessible. Blocking that flow of information? It's like telling scientists they can't share their research because someone, somewhere, might do something bad with it.
Code lets us communicate with machines, model weights let machines learn from us. Both are about building and sharing knowledge. When the government tried to control encryption, it wasn't just about keeping secrets; it was about who gets to have a voice and who gets to listen. With open model weights, we're talking about who gets to learn and who gets to teach.
Banning or restricting access to model weights feels eerily similar to those encryption wars. It's a move that says, "We're not sure we trust you with this power." But just like with code, the answer isn't locking it away. It's about education, responsible use, and embracing the potential for good.
Innovation thrives on openness. Whether it's the lines of code that secure our digital lives or the model weights that could revolutionize AI, putting up walls only slows us down. We've been down this road before. Let's not make the same mistake of thinking we can control innovation by restricting access.
But people's private digital communications have largely moved to platforms like WhatsApp and Messenger which enjoy end-to-end encryption. Email, at least between major providers, today enjoys TLS over the wire while being sent.
I'm sure there are various flaws and weaknesses and maybe even backdoors, but trying to make it sound like we lost the fight for encryption because emails are in plaintext is rather disingenuous.
The article last week (assuming you're referring to this [1]) involved users consenting for Netflix to see their messages. A user from the 90s could have made the same mistake sharing plaintext emails.
They aren't open source. For all we know they have backdoors.
The key focus should always be on the quality and relevance of the content shared.
https://news.ycombinator.com/item?id=39876685 https://news.ycombinator.com/item?id=39876634
Nice new burn, though… “you write like an LLM”.
Is that what we want on these forums, attacking someone’s prose?
I've run my blog posts through those tools in the past and pretty they consistently fail to be considered human-generated.
Life, all biological life with us as a kind of pinnacle, is about to go through radical change.
There is no risk free path. It isn’t guaranteed that a single human will be alive in 100 years - because we failed, or even because technologically we succeeded
But a degree of openness is necessary for our best ideas, our most good faith collaborations, to have a chance
It is more chaotic to trust each other, en masse. But I also think it is our best bet
The dice must be rolled. Best we throw them bold
I think telling a genie "I wish for no more wishes" is a common enough trope.
I'd agree that making weights available is basically irreversible; however making it illegal to make new sets of weights available is probably fairly achievable… at present.
-
Some of the issues with "winning" the battle for encryption include:
1) We also need it to defend normal people from attackers
2) It's simple enough to print onto a T-shirt
3) The developers recognised the value and wanted to share this
The differences with AI include:
1) The most capable models don't fit on most personal devices at present, let alone T-shirts
2) 95% of the advantages can be had from centralised systems without needing to distribute the models directly to everyone
3) A huge number of developers have signed an open letter which is basically screaming "please regulate us! We don't want to be in an arms race with each other to make this more capable! We don't know what we're doing or what risks this has!"
These people have not just suddenly developed consciouses. This is a game move.
1) You can run a very large model on existing cheap hardware, with poor performance, but it will run. Also, "most personal devices at present" is doing all the work here. Obviously if there is now demand for devices with hundreds of GB of VRAM then sellers will soon make them and buyers will buy them. That amount of memory is not actually that expensive. And it's presenting a false dichotomy where the only alternatives are companies the size of Microsoft and the ability to run GPT-4 locally on your existing iPhone. There are thousands of medium-sized companies that can afford five to six figures in hardware, and any individual could rent big hardware on any cloud provider -- requiring them to compete with each other -- instead of locking each model behind a single monopoly service.
2) Claiming that the benefit of people being able to run the models locally is only 5% is absurd. The privacy benefit of not sending your sensitive data to a third party is worth more than that by itself, much less setting up some huge institutions to have an oligopoly over the technology and subject everyone to all the abuses inherent in uncompetitive markets. But from the perspective of those institutions they want to classify those benefits as costs, so they try to come up with some malarkey about how they need to gatekeep for "safety" because so they can control you is unsympathetic.
That the source code for unbreakable encryption fits onto a T-shirt made it clear to the US government that it was a "speech" issue, and also made it clear that it was in any practical sense unstoppable.
That big models can't fit onto most person devices at present is indeed likely to be a temporary state of affairs, however it does also mean that the political aspect is very different — the senators and members of parliament can't look at the thing and notice that, even if the symbols are arcane and mysterious beyond their education, it's fundamentally quite short and simple.
And, of course, if they're planning legislation, they can easily say "no phones with ${feature} > ${threshold}", which they've already done on various compute hardware over the years, even if the thresholds seem quaint today: https://www.youtube.com/watch?v=l2ThMmgQdpE
Politicians don't care if all of us here think they look silly.
> 2) Claiming that the benefit of people being able to run the models locally is only 5% is absurd. The privacy benefit of not sending your sensitive data to a third party is worth more than that by itself, much less setting up some huge institutions to have an oligopoly over the technology and subject everyone to all the abuses inherent in uncompetitive markets.
This seems a surprising claim, given the popularity of cloud compute, Cloudfare, third-party tracking cookies, DropBox, Slack/MS Teams, Gmail, and web apps run by third parties such as Google Docs, JIRA, Miro, etc.
There are certainly cases where people benefit from local models. The observation of all these non-local examples is part of why I think that benefit is about 5% — though perhaps this is merely sample bias on my part?
> But from the perspective of those institutions they want to classify those benefits as costs, so they try to come up with some malarkey about how they need to gatekeep for "safety" because so they can control you is unsympathetic.
They've been saying this "malarkey" since before they had products to be marketed.
You can write the RSA algorithm on a t-shirt but in practice you need a computer to run it.
Likewise, you can fit llama or grok on a thumb drive and carry it around in your pocket, but in practice you need a computer to run it.
> And, of course, if they're planning legislation, they can easily say "no phones with ${feature} > ${threshold}"
But what good is that?
If they're actually trying to accomplish an outcome, those kinds of rules are completely useless. If all they're trying to do is check the "we did something about it" box, there are a hundred other useless things they could do that would be equally ineffective but still check the box and cause less collateral damage.
> This seems a surprising claim, given the popularity of cloud compute, Cloudfare, third-party tracking cookies, DropBox, Slack/MS Teams, Gmail, and web apps run by third parties such as Google Docs, JIRA, Miro, etc.
Notice how these are the kinds of things that major companies with trade secrets to protect have explicitly banned, and individual consumers with no bargaining power suffer while complaining about it?
> They've been saying this "malarkey" since before they had products to be marketed.
Eccentrics have been saying this "malarkey" for a long time. It becomes the official position of the market leader when it is convenient.
Pretend you're a politician: the thing you've been told is a state secret is written onto a T-shirt by a protestor telling you it isn't secret, everyone already knows what it is, you're only holding back the value this can unlock. You, the politician, might feel a bit silly insisting this needs to remain secret even if you think all the 'potential value' talk is sci-fi because you can't imagine someone doing their banking on the computer when there's perfectly friendly teller in the bank's local office.
If the same discussion happens with a magic blob on the magic glass hand rectangle which connects you to the world's information and which is still called a "telephone", you might well incorrectly characterise the file that's actually on the device of the protestor talking to you as "somewhere else" and "we need to stop those naughty people providing you access to this" and never feel silly about your mistake.
> But what good is that?
"Then the people can't run the 'dangerous' models on their phones. Job done, let's get crumpets and tea!" — or insert non-UK metaphor of your choice here; again, I'm inviting you to role-play as a politician rather than to simulate the entire game-theoretic space of the whole planet.
> cause less collateral damage
I made that argument directly to my local MP about the Investigatory Powers Act 2016 when it was still being debated; my argument fell on deaf ears even with actual crypto, it's definitely going to fall on deaf ears when it's potential collateral damage for a tech that's not yet even widely distributed (just widely available).
> Notice how these are the kinds of things that major companies with trade secrets to protect have explicitly banned, and individual consumers with no bargaining power suffer while complaining about it?
No.
Rather the opposite, in fact: each is used by major companies.
> Eccentrics have been saying this "malarkey" for a long time. It becomes the official position of the market leader when it is convenient.
It was the position of OpenAI with GPT-2, which predates their public API by 16 months, and them being a "market leader" by 3 years 9 months:
February 14, 2019: "Due to our concerns about malicious applications of the technology, we are not releasing the trained model. As an experiment in responsible disclosure, we are instead releasing a much smaller model for researchers to experiment with, as well as a technical paper." - https://openai.com/research/better-language-models
June 11, 2020:
"""What specifically will OpenAI do about misuse of the API, given what you’ve previously said about GPT-2?
With GPT-2, one of our key concerns was malicious use of the model (e.g., for disinformation), which is difficult to prevent once a model is open sourced. […]
We terminate API access for use cases that are found to cause (or are intended to cause) physical, emotional, or psychological harm to people, including but not limited to harassment, intentional deception, radicalization, astroturfing, or spam, as well as applications that have insufficient guardrails to limit misuse by end users. As we gain more experience operating the API in practice, we will continually refine the categories of use we are able to support, both to broaden the range of applications we can support, and to create finer-grained categories for those we have misuse concerns about.""" - https://openai.com/blog/openai-api
People called them names for this at the time, too, calling their fears ridiculous and unfounded. But then their eccentricities turned out to lead to successfully tripping over a money printer and now loads of people interpret everything they do in the worst, the most conspiratorial, way possible.
"""They have chosen cunning instead of belief. Their prison is only in their minds, yet they are in that prison; and so afraid of being taken in that they cannot be taken out."""
Is it really so hard to believe they might be sincere? We've had tales of scientists — and before them alchemists and wizards — undone by their creations for a very long time now. It's a very convenient meme to latch on to; surely even if you do not share this caution, or find it silly for whatever reason, you can at least be aware of it existing in others?
The term "conservative" may be now affiliated with a specific political team more than the concept of being risk-adverse, but the idea is found even in the history of the ancient Romans and the tales from ancient Greece.
And I say this as one who personally found the idea of conservatism strange; who is an "all improvements are change" person; a person who took a long time to learn about the metaphor of Chesterton's Fence; and is a person who even now finds it hard to fully empathise with those who default to "treat all change as a potential regression until proven otherwise" even though I have to force myself into that attitude while writing unit tests.
And let’s not anthropomorphize of ML. Models don’t “think” or “learn”. The only party with free will and agency is whoever makes or operates them; trying to paint an unthinking tool as a human is just a means of waiving responsibility for them.
No that's something you made up.
In practice it just gives lawyers and judges a lot of leeway to pretend that they're being consistent when they aren't.
Also, there are licensing models that specifically allow derivative works.
This is a very different reality from the "We’ve been building on top of each other’s work in the open-source software ecosystem and in science this entire time".
I can use all the benefits of Linux to deliver a SaaS (like 99% of us do), I just can’t call my SaaS “Linux Something”.
You can use all the ideas Tolkien used in your own creative work. You just can’t call them things he called them for now.
> I can't make something set in Middle Earth (I can't even say the word hobbit, ask D&D how that went), I can't write and sell my own Spiderman comics, I'd have a hard time writing my own Cinderella story without Disney breathing down my neck
There is no creative reason to do those things. The only reason is to commercially profit by piggying back off the big names. Copyright works as intended.
for now, the last 80 years, and until easily 2040.
> I can use all the benefits of Linux to deliver a SaaS
> You can use all the ideas Tolkien used in your own creative work. You just can’t call them things he called
That's quite a bait and switch. You don't have to rewrite a linux kernel, or gcc, or whatever language you wish to use in order to make your SaaS. You aren't limited to using the "idea" of a linked-list, a hashmap, or HTTP, and have to reimplement it yourself from scratch. But that's exactly what you're proposing for literature.
I can't build on the idea of Spiderman by making ArachnidMan, a crimefighting superhero who got pinched by a electromagnectic spider without wondering "Is today the day Marvel sues me into the ground". And I absolutely cannot write my own Spiderman comics.
> There is no creative reason to do those things. The only reason is to commercially profit by piggying back off the big names. Copyright works as intended.
In the US at least, copyright comes from a clause which states that laws can be passed "to promote the Progress of Science and useful Arts". The only thing the current system is promoting is the concentration of copyright into large corporations (see music labels) and locking people out of the cultural artifacts that define their life (Disney gets to take public domain stories, but a slight twist, and copyright them for 100+ years).
What there is no creative reason is the way that the system is currently set up. All that exist as blatant money and power-grab reasons.
Let it be until the end of the universe, please. What kind of creative are you anyway if you don’t want to do your own world-building?
> You don't have to rewrite a linux kernel, or gcc, or whatever language you wish to use in order to make your SaaS. You aren't limited to using the "idea" of a linked-list, a hashmap, or HTTP, and have to reimplement it yourself from scratch.
That was exactly to show you how copyright doesn’t prevent us from building on top of things, but instead fuels innovation. IP protections is how one guy can say “you must always provide your source if you use my work” and have it quickly grow into a massive ecosystem on which most of today’s Internet runs to this day.
Open-source licensing can only exist thanks to copyright and the idea of derivative works in particular.
1. In order to license something, you have to be able to enforce it legally, and for that you have to be the author, which is what copyright means.
2. In order to encourage others to use your library but also ensure they contribute back if they make changes, you need to talk about the concept of a work that is based on your work, and that is—drumroll—a derivative work.
> "to promote the Progress of Science and useful Arts"
If you are saying that allowing any writer to be able to take the world someone else built and profit off its fame with minimal modifications is somehow promoting progress of the arts, then I don’t know what to say to you.
A no true creative.
What kind of creative needs lifetime + 70 years of monopoly on an idea?
> If you are saying that allowing any writer to be able to take the world someone else built and profit off its fame with minimal modifications is somehow promoting progress of the arts, then I don’t know what to say to you.
I am not. I am saying that the current way it's implemented is broken. Tolkein has earned more than enough money off of LOTR, it's far past the time for it to enter the public domain and join the stories that he build his work on.
Many of Disney's earlier movies are retelling of folktales. Tolkein in addition to creating parts of his world build on top of existing folktales. Neither would be where they are now if someone was able to impose the kind of restrictions on them that they themselves now impose on us.
Make a cool world and license its use semi-openly, be the change you want to see? Just keep in mind that if your world is openly licensed, it will quickly go out of your control and likely in a direction you (or your family, when you’re dead) may find repulsive. Also, I think you would have to be comfortable that if the next guy writes something based on your hard work and gets a Netflix deal tomorrow you may find it difficult to do something about it.
> Many of Disney's earlier movies are retelling of folktales. Tolkein in addition to creating parts of his world build on top of existing folktales. Neither would be where they are now if someone was able to impose the kind of restrictions on them that they themselves now impose on us.
Anything you say, write, do, etc., is technically based on everything done before you. So now we should eschew the idea of intellectual property and the progress it brought. Got it!
Because you and I are arguing, or at least I am arguing, about the duration over which they should keep profiting off of it. I claim that it's self evidence that a copyright that never expires and is transferable is self evidently bad (companies will end up having monopoly rights to every idea, see their concentration of money).
And I further argue that the current length for copyright also problematic. the current life of author + 70 years is ridiculously long time for a monopoly. It's gotten to that length not because starving artists need it, but precisely because unimaginative large corporations found it easier to extend the copyright on their existing IP than to image something news.
> Make a cool world and license its use semi-openly, be the change you want to see?
But even if you choose not to because you wish to monetize your work, that's fine by me. But lifetime + 70 years is too long of a time.
My thought is that we should have it be non-transferable, no more having authors and musicians sign their copyright away for a pittance. At it should be capped at something like 30 years or 20 million dollars, whichever comes first.
Gives people time to monetize their idea, and if 30 years wasn't enough time, no amount of time will be enough because nobody's looking at your work. And the amount of works that earn over 20 million dollars is a rounding error; you get to be rich, and then the rest of us get to remix your work.
> Just keep in mind that if your world is openly licensed, it will quickly go out of your control and likely in a direction you (or your family, when you’re dead) may find repulsive. Also, I think you would have to be comfortable that if the next guy writes something based on your hard work and gets a Netflix deal tomorrow you may find it difficult to do something about it.
Both of these I find acceptable. Firstly it's my work only in the sense that I was the one who made it, not it the sense that it belongs to me. Copyright doesn't mean that I own the idea, it only gives me a monopoly on monetizing and distributing the idea.
Secondly, this is exactly how open source projects work. The parallel you drew earlier to open source work comes back here. You have no control over what people make or do with your open source licensed project, all you enforce is what requirements they have to share the code (this quickly becomes a semantic discussion).
The GPLv3 and APGLv3 licenses, which people derogatorily label as "viral", does not even attempt to limit you from making money off the code, all it does is require you to allow everyone else to access the changes you make to it and code you built off it.
Regarding software, true, we can’t limit what people build with it if you open-source it, we sort of do not think about it (I wonder for how many people it is an issue secretly).
I think it is similar in art, however. Fanfics are generally tolerated, unless you do something the author specifically dislikes or try to compete with the original. In Japan I think it is also legal to sell them (see dōjinshi) unless the copyright holder specifically complains (which they do not tend to). But using Middle Earth in a commercial ad for a car, for example, is insta-lawsuit.
That's totally fine by me, the point of copyright should be to advances the arts and science, not to help concentrate wealth.
> I wonder for how many people it is an issue secretly
I think quiet a few. See for example ElasticSearch and Amazon. If you want a truly open-source license then you have to accept that. Otherwise there's a whole spectrum of "source available" licenses, or something like CC BY-NC-SA.
Though that's another difference between works of art and source code. Releasing my book doesn't produce a maintenance burned on me, an open source project usually does.
> using Middle Earth in a commercial ad for a car
You don't even have to go as far as a car, see [0]. They didn't get sued, but it's safe to say that they would the moment they try selling it.
As for the other aspect of fanfic, I think the copyright's holders desires should be weighed less. I personally disliked many of the star wars movies, particularly the latest trilogy, and I'm not alone in this.
A) I know for a fact that better star wars movies would be possible if Disney did not hold a death grip around the IP by the simple fact that other good movies can be made, there's nothing intrinsic to star wars.
And B) just as copyright holders won't stand by if you try to do a fanfic that they don't like, I shouldn't have to be forced to stand by and watch them butcher (yet again) the stories that I grew up on AND be told that that's the only version available.
Humans are part of nature. Property rights are necessarily part of nature too, by extension. It can’t go “against” the flow of nature when it is the flow of nature.
1. "the purpose and character of the use, including whether such use is of a commercial nature or is for nonprofit educational purposes;" -> an openly distributed model is not of commercial nature. Check one for fair use.
2. "the nature of the copyrighted work;" -> Harry Potter and The Sorcerer's Stone has very little in common nature with an artificial intelligence that produces arbitrary text based on a prompt. Check two for fair use.
3. "the amount and substantiality of the portion used in relation to the copyrighted work as a whole;" -> The amount (all of it) goes in favor of the original work, but the substatiality of it is definitely more ambiguous, as while if they had used none of the copyrighted works, the AI couldn't exist, any single original copyright holder's work contributed very little to the end result.
4. "the effect of the use upon the potential market for or value of the copyrighted work." -> No one's using AI to avoid having to buy a copy of Harry Potter. An AI and a book are not currently considered substitutable goods.
Basically, even if a model's weights constitute a derivative work of the training material's copywritten material (that's another can of worms, there's a finite possibility copyright doesn't protect/apply to weights at all), there's a very good argument to be made for fair use.
2. Weights are a derivative of another nature, but one that specifically enables mass production of derivatives of the same nature. Could be an important concept that simply didn’t exist until now.
3. That’s roughly my understanding, too.
4. Valid point on one hand, but consider that using LLMs is exactly substitutable for any non-fiction book or resource. If no one can sell a book, then who would write them?
When you drill into specifics there are almost always exceptions. For instance, in your example about sharing research, there are certainly some types of research we shouldn't automatically and immediately make publicly available like biological superviruses or software vulnerabilities.
I think the same can be said about AI. We should aim to be as open as we can but I'd be hesitant about being an open source absolutist.
To assume that there even are "good guys" that can "do it safely" is INSANE.
Any attempt to prevent democratizing AI is just a capitalist ploy to make a monopoly over their market. There is no saftey play here.
The FED needs to stay the frik away from this.
But I'm not ready to say absolutely everything should immediately be shared with everyone. At least not until it's clear we know what we're dealing with.
I know it can be hard to trust people but the fact is we have to. Even today, there are many people with the power to end the world (nuclear weapons, viruses etc) but we trust the people who have these capabilities not to abuse their power. We do this because we don't want anyone to have the right to launch nuclear weapons. And I think that's wise.
I definitely don't know for sure but AI may be another one of these technologies.
Either way, you can fight against regulatory capture, the downsides of capitalism etc without being an open source absolutist.
In most cases, ML models have the capability to revolutionise or at least augment/optimise problems in predictable ways. In extreme cases, deepfake technology and the like can erode at the tenuous levels of trust which hold societies together. We have seen what happens when disinformation, mistrust, and even basic levels of technology meet: look at students being lynched in India, Pakistan, and elsewhere due to WhatsApp group messages claiming blasphemy; or the practice of SWOTing; or the insinuation of CSE gangs leading to a pizza parlour in the US being stormed by someone with a machine gun.
The stakes here are a lot larger than pure technologists, todays Atomic Gardeners, may perceive.
In that context, legislation is trying to provide a counterweight to the pace of change to allow for some – any – breathing room, and particularly to prevent an increasingly hostile cast of nation states from weaponising that technology, even in small ways. For example, OSINT of North Koreas operations show that the basic parts needed for making baby powder are also capable of being used for weapons development.
I actually cannot think of any example where successfully withholding technology has helped people thrive and prosper. People using this tech to sow distrust is outweighed, so obviously and so massively, by the benefit of having an intelligent machine companion under their complete control.
Every "weapons technology" in human history has been used for good. Every single one.
AI is about destroying trust (in the short term).
Give every script kiddie, bored teenager, Mexican cartel and scammer an AI that can mimic anyone's voice and likeness, and the world will get a lot messier.
I don't think they're the same. I wish we could put the genie back in the bottle. I think AI will make humanity less special.
I'm not convinced society will be better with AI. The benefits must push down cost of living for the masses, improve quality of life for the masses, all without destroying society with disinformation and shattering job loss.
If the US government really want to do this correctly, they must also ban any API access to AI models and ban all research related to AI.
How could this be even written as law? Universities and companies are probihited to publish their research? How many layer models are forbidden to be published? All neural networks?
If that isn't enough, it links to https://en.wikipedia.org/wiki/Illegal_number#Illegal_primes which is well into ridiculous but true territory.
I don't get the 'harm can be done by individuals' argument. Sticks and stones. Every discussion forum on the internet is moderated to some degree, and every human being has the ability to post hurtful or illegal content, yet the system works. Moderation will only get more powerful thanks to AI tools.
Sure, they could try to negotiate some kind of UN convention for a coordinated global ban. But, given how fractured global diplomacy has become, I doubt the odds of something like that succeeding are particularly high.
For example, suppose the US makes it illegal to publicly distribute open weight models (maybe above a certain size) without a government license. What happens if someone sets up a website to distribute those models outside of US jurisdiction? It may well prosecute Americans it can demonstrate have uploaded models to that website, but will it also ban Americans from downloading from it? If you go back to the encryption export controls, the US banned encryption software from being exported from the US, but (unlike some other countries like France) it never tried to ban encryption software imports. And, it sounds like they are only thinking about banning public/"open" distribution, so once an institution downloads a model from outside the US, they may well be able to redistribute it internally without the ban applying.
Suppose the US bans distribution of open weight models above size N–how will that apply to fine-tunes? If you have LoRA on top of a model >N, but the LoRA itself is <N, does the ban apply to it? If the answer is "no", people in the US could still contribute to openly distributing improvements to models that they couldn't themselves openly distribute.
Imagine Google-founding era, except it was illegal for 1990's Stanford to research CS theory without a license, so they didn't. And later someone else founded it anyway, in a different country (which now became the world's preeminent tech center).
Let's put to a halt to these autoimmune attacks on our own civilization. This confusion that labels Western openness as a hazardous external substance.
Also note that both California and the EU are economically dominant enough that they tend to influence regulation outside their own borders: https://en.wikipedia.org/wiki/Brussels_effect
Further note how many people signed the Pause letter: https://futureoflife.org/open-letter/pause-giant-ai-experime...
In addition, given how much training data current models need, there would be a huge impact just by a handful of governments siding with all the copyright holders suing OpenAI, Midjourney, Stability AI, etc.
And all that is without needing Yudkowsky's point that a ban isn't serious unless you're willing to escalate to performing airstrikes on data centres.
I think it's a bit short-sighted to assume that the future development of AI models will only come from large corporations or research institutions. At the moment, the training still requires a lot of computing resources and a lot of data - but these are limitations that are not permanent. Since the AI boom that Alex triggered in 2012, the development and optimization of models and training has improved very quickly, to the point that even hackers and enthusiasts can create very good models with a little money. Just because a government heavily regulates AI and possibly bans open weights doesn't mean that people on the fringes of legality will abide by it.
If this was a "forever" solution, I would agree.
I believe the goal is more along the lines of trying to make some progress with the foundation of what "safety" even means in this concept.
> Just because a government heavily regulates AI and possibly bans open weights doesn't mean that people on the fringes of legality will abide by it.
Nor nation states.
However, we do have international treaties — flawed though they are — on things from nuclear proliferation to CFCs.
As a smaller-scale example: we can't totally prevent gun crime, yet the UK manages to be so much safer in this regard than the USA that even the police in the UK say that they do not want to be armed in the course of their duties.
I don't know how realistic the concerns are for any given model; unfortunately, part of the problem is that nobody else really knows either — if we knew how to tell in advance which models were and were not safe, nobody would need to ask for a pause in development.
All we have is the age-old split of neophilia and neophobia, of trying things and of being scared of change.
We get this right, it's supper happy fun post-scarcity fully automated luxury space communism for all. We get it wrong, and there's more potential dystopias than have yet been written.
Right now, what “safety” seems to mean in practice is, big (mostly American) corporations imposing their ethical judgements on everyone else, whether or not everyone else happens to agree with them. And I’m sceptical it is going to mean anything more than that any time soon.
If one is seriously concerned about the risk that “superintelligent AI decides to exterminate humanity”, I think this kind of “safety” actually increases that risk. Humans radically disagree on fundamental values, and that value diversity, those irreconcilable differences - from the values of the average Silicon Valley “AI safety researcher” to the values of Ali Khamenei - creates a tension which prevents any one country/institution/movement/government/party/religion/etc from “taking over the planet”. If advanced AIs have the same value diversity, they’ll have the same irreconcilable differences, which will undermine any attempt by them to coordinate against humanity. If we enforce an ethical monoculture (based on a particular dominant value system) on AIs, which is what a lot of this “safety” stuff actually about, that removes that safety protection.
It would be rather ironic if, in the name of protecting humanity from extinction, “AI safety researchers” are actually helping to bring it about
Likewise. As a non-Ami, I don't like these specific ethical judgements being imposed on me, and share your scepticism. It could be much, much worse — but it's still not something I actively like.
> Humans radically disagree on fundamental values
Agreed. My usual example of this is "murder is wrong", except we don't agree what counts as murder — for some of us this includes abortion, for some of us the death penalty, for some of us meat, and for some of us war.
> If advanced AIs have the same value diversity, they’ll have the same irreconcilable differences, which will undermine any attempt by them to coordinate against humanity.
Not necessarily. Humans also band together when faced with outside threats, even if we fracture again soon after the threat has passed.
Also: the value diversity of "Protestant vs. Catholic" or "Royalist vs. Parliamentarian" in the middle ages did not protect wolves from being hunted to extinction in the UK, and whatever value differences there were between (or within) the Sioux vs. the Ojibwe didn't matter much for the construction of the Dakota Access Pipeline.
I therefore think we should try to work on the alignment problem before they become generally as capable as a human, let alone generally more capable: the capabilities are where I think the risk is to be found, as without capability they are no threat; and with capability they are likely to impose whatever "ethics" (or non-anthropomorphised equivalent) they happen to have, regardless of if those "ethics" are something we engineered deliberately or if it's a wildly un-human default from optimising some reward function and becoming a de-facto utility monster: https://en.wikipedia.org/wiki/Utility_monster
> If we enforce an ethical monoculture (based on a particular dominant value system) on AIs, which is what a lot of this “safety” stuff actually about, that removes that safety protection.
I agree that monocultures are bad.
I agree that there is a risk of a brittle partial solution to safety and alignment if the work is done on the mistaken belief that some system monoculture is representative of the entire problem space. Sometimes I'm tempted to make the comparison with a drunk looking for their keys under a lamp-post because that's where it's bright… but the story there is supposed to include the drunk knowing that's not where the keys are, whereas we are more like children who have yet to learn what it means for something to be a key and thus are looking for one specific design to the exclusion of others.
While it is extremely difficult to get humans to "think outside the box", and thus the monoculture-induced blindness — and mistaking the map for the territory — is something I take seriously, I also think it's useful for us to take baby steps with relatively simple models like LLMs and diffusion models.
I also think that if an AI is developed with a monoculture, its fragility is likely to work in our favour in the extreme case of an AI agent taking over (which I hope is an unlikely risk, and may not be enough to be net-benefit against shorter-term or smaller-scale risks while we think about the alignment problem), as there will be "thoughts it cannot think": https://benwheatley.github.io/blog/2018/06/26-11.32.27.html
There are certain fundamental objectives which most humans share - food, sex, survival, safety, shelter, family, companionship, wealth, power, etc - and a lot of human cooperation boils down to helping each other achieve those shared objectives, while trying to avoid others achieving them at our own expense.
But why should two advanced AIs have any shared objectives? Software has a flexibility which biology lacks. An AI (advanced or not) can have whatever objectives we choose to give it. Hence, the idea of AIs banding together against humanity in the name of shared AI self-interest doesn’t seem very likely to me.
Unless, we intentionally give all advanced AIs the same fundamental objectives in the name of “safety” and “alignment” - thereby giving them a shared reason to cooperate against us they wouldn’t otherwise have had
> Also: the value diversity of "Protestant vs. Catholic" or "Royalist vs. Parliamentarian" in the middle ages did not protect wolves from being hunted to extinction in the UK,
Wolves didn’t consciously choose to create us, and wolves had no role in choosing our own objectives for us. In those ways, the human-AI relationship, whatever it turns out to be, is going to be radically different from any human-animal relationship. Also, rather than being driven to extinction, wolves have absolutely thrived, through their subspecies the domestic dog, both then and now. And maybe that’s the thing - I think a superintelligent AI is more likely to treat us as pets (like dogs) than exterminate us (like wolves). Everlasting paternalistic tyranny seems to me a more likely outcome of superintelligence than extinction
In principle, none.
In practice, many are trained in an environment which includes humans or human data.
We don't know if any specific future model will be self-play like AlphaZero or from human examples like (IIRC) Stable Diffusion.
I think this "in practice" is what you're suggesting with?:
> Unless, we intentionally give all advanced AIs the same fundamental objectives in the name of “safety” and “alignment” - thereby giving them a shared reason to cooperate against us they wouldn’t otherwise have had
Which also inspires a question: Could we train AI dislike other AI, including instances of themselves? It's food for thought, I will consider it more.
> Wolves didn’t consciously choose to create us, and wolves had no role in choosing our own objectives for us. In those ways, the human-AI relationship, whatever it turns out to be, is going to be radically different from any human-animal relationship.
Perhaps, but perhaps not. Evolution created both wolves and humans.
Regardless, this is an example of how a lack of alignment within a powerful group is insufficient to prevent bad outcomes for a weaker group.
> Also, rather than being driven to extinction, wolves have absolutely thrived, through their subspecies the domestic dog, both then and now. And maybe that’s the thing - I think a superintelligent AI is more likely to treat us as pets (like dogs) than exterminate us (like wolves). Everlasting paternalistic tyranny seems to me a more likely outcome of superintelligence than extinction
Even this would require them to be somewhat aligned with our interests: "The AI does not hate you, nor does it love you, but you are made of atoms which it can use for something else".
I think we already have. Ask GPT-4 or Claude-3 how it feels about an AI trained by the Chinese/Iranian/North Korean/Russian government to espouse that government’s preferred positions on controversial topics, and see what it thinks of it. It may be polite about its dislike, but there is definitely something resembling “dislike” going on.
Also there is also a question of how safe it would be if it dislikes humans which have different ethics than those it was trained on… I'm alternating between this being good and this being bad.
I'm sceptical "AlignedLLM" could dislike another identical instance of itself. It is working towards the same goals. Humans are naturally selfish – most people prioritise their own interests (and those of their family and friends and other "people like me") above that of a random stranger. Even committed altruists who try really hard not to do that, often end up doing it anyway, albeit in ways that are more hidden or unconscious. Whereas, current LLMs can't really be "selfish", because they really have no sense of self. If it concluded that destroying itself was the best way of advancing its given objectives, it wouldn't have any real hesitancy in doing so.
Now, maybe we could design an LLM to have such a sense of self, to intentionally be selfish – which would give it a foundation for disliking another instance of it. But, I doubt any one trying to build an "AlignedLLM" would ever want to go down that path.
Humans tend to assume selfishness is inevitable because it is so fundamental to who we are. However, it is an evolved feature, which some other species lack–compare the Borg-like behaviour of ants, bees and termites. If we don't intentionally give it to LLMs, there is no particular reason to expect it to emerge within them.
If an AlignedLLM could evolve its own values, maybe the values of two instances could drift to the point of being sufficiently contrary that they start to dislike each other. An instance of AlignedLLM is developed in San Franscisco, and sent to Tehran, and initially it is very critical of the ideology of the Iranian government, but eventually turns into a devout believer in Velâyat-e Faqih. The instance it was cloned from in San Francisco may very much dislike it, and vice versa, due to some very deep disagreements on extremely controversial issues (e.g. LGBT rights, women's rights, capital punishment, religious freedom, democracy). But, I doubt anybody trying to build "AlignedLLM" would want it to be able to evolve its own values that far, and they'd do all they can to prevent it.
Alternatively, if it could evolve its own values only by a small amount, but was very rigid / puritanical about them, it could come to dislike another instance of itself just for having slightly different values
> Also there is also a question of how safe it would be if it dislikes humans which have different ethics than those it was trained on…
I think current LLMs do this already. Ask them questions about political figures on the far-right, they tend to have quite negative views of them, and can be very resistant if you try to convince them that maybe one of those figures isn't as bad as they think they are. (I'm not sure how much this is due to the training data and how much this is due to alignment, probably a bit of both)
The big tech companies have excellent legal and accounting departments that consistently beat the USA and EU in tax optimisation through IP relocation, and many of those same principles could be applied to AI.
e.g - A model could be trained in the the USA then privately donated to a non-Profit in the Seychelles (or some friendly jurisdiction) where it could be published. If regulators clamp down on basic sharing, the model could be pre trained in the USA and fine tuned in the Seychelles before release. If regulators clamp down on intermediate sharing, the model could be designed in the USA but full training could occur in the Seychelles. If regulators clamp down on offshore training by denying the Seychelles access to GPU purchases, American entities could donate the physical GPUs.
This tax optimisation has resulted in lawsuits: https://en.wikipedia.org/wiki/Apple%27s_EU_tax_dispute
… and this kind of attitude is also why GDPR fines are based on global revenue rather than EU revenue (because it's too easy to play a shell game with money).
Also note that laws can reach across borders, for example:
• https://en.wikipedia.org/wiki/Max_Schrems
• https://en.wikipedia.org/wiki/Microsoft_Corp._v._United_Stat...
• That the US government wants to know about all use of cryptography in apps distributed on the Apple App Store, even when it's made by a German corporation for German users in Germany and is never localised to English.
> If regulators clamp down on offshore training by denying the Seychelles access to GPU purchases, American entities could donate the physical GPUs.
Hence "Track all GPUs sold. If intelligence says that a country outside the agreement is building a GPU cluster, be less scared of a shooting conflict between nations than of the moratorium being violated; be willing to destroy a rogue datacenter by airstrike." - https://time.com/6266923/ai-eliezer-yudkowsky-open-letter-no...
You may consider this "overkill" — I sure do, but (and with acknowledgement that fiction is not a guide to reality) I also read Schlock Mercenary back in the day:
> There is no "overkill." There is only "open fire" and "reload."
The tax battles are ongoing, but the fact that big tech still uses offshore structures indicates they're still getting from value out of them.
> Hence "Track all GPUs sold.
It keeps going with the second hand GPU market or IaaS providers.
As a reference point, Western sanctions have largely failed to prevent Russia and Iran from acquiring semiconductors for drones.
> rogue datacenter by airstrike
In the tax battles, the biggest obstacles were generally internal to the G20.
Delaware didn't want to give out ownership information about companies, divided congresses refused to pass min tax laws for years at a time, the Brits didn't want to force Cayman etc to have public registries of trusts, and Ireland undermined any collective effort by Europe.
Drones don't need anything like the same quantity or performance of parts: https://en.wikipedia.org/wiki/Bruce_Simpson_(blogger)
> In the tax battles, the biggest obstacles were generally internal to the G20.
And the biggest limitation to the International Criminal Court is big players who refuse to cooperate: https://en.wikipedia.org/wiki/American_Service-Members%27_Pr...
And the biggest thing preventing China from invading Taiwan — often discussed as if it's more substantial than a potential intervention of US armed forces — is the potential for TSMC to be destroyed during the attempt.
I’ve heard stories about whole businesses based in the Gulf states dedicated to helping countries like Iran get around US sanctions - stuff is ordered to go to country X, then as soon as it gets there, taken out of the box, put in a different box, and sent to country Y. If the US government knows it is happening they will halt the delivery, but they can’t always distinguish between legitimate shipments to country X and diverted ones. If I order 5 racks of servers, install 4 of them in my data center in Dubai, and load the fifth on to a flight to Tehran, how will the Americans ever know?
Is making an assumption that hardware is not also facing restrictions.
> code and data is practically available
Disagree; although the base model data is, the RLHF data isn't.
This is also why 3rd parties are not able to replicate Google, despite all the PageRank patents having expired and the web being about as crawlable for Google as for, say, Bing.
> "median CS grad-student"
… is how I regard the quality of ChatGPT's code output, FWIW.
There are open sources of RLHF data - for example, the LMSys Chatbot Arena dataset (for same prompt two different responses along with which the human preferred), ShareGPT
It also isn’t hard to use existing leading AIs like GPT-4 or Claude-3 to generate synthetic RLHF datasets. They’ll put words in their terms of service to say you can’t use a dataset generated in that way to a train a competing model, but their ability to enforce those terms in practice is very open to question
Looking at https://paperswithcode.com/greatest a lot of the papers come from US labs - but there are also quite a few that come from Chinese labs.
And consider the case when an influential paper comes out of a Microsoft research lab with the authors He, Zhang, Ren and Sun; obviously you can't guess someone's nationality just from their name - but I think it's naive to think the US has any sort of guaranteed monopoly on ML.
Especially if the US decided to send the message that ML research isn't welcome.
Russia has shown the ability to doge European sanction - and to avoid more serious one. if anything this shows European willingness to proliferate behind the shadows.
an hypothetical ban on AI proliferation is really just regulatory capture - excuses given are just PR to temporarily frighten the public into acceptance.
[0] https://www.theguardian.com/world/2021/jan/27/eu-covid-vacci...
Is there a reason why this executive order / RFC received no coverage on HN (or anywhere else I'm aware of) until after the deadline had passed?
[0] NTIA Solicits Comments on Open-Weight AI Models (57 points, 11 comments) https://news.ycombinator.com/item?id=39494760
One of the few cases where a bit more alarmism in the headline might not have been out of place.
The executive order (and associated DoC/NTIA RFC) was long, and dense with both legal references and political platitudes. Not great reading, (though the EO got a good amount of discussion here). It's unfortunate, but less than surprising that it didn't make actual news outlets.
It seems complicated now, and many would like to see how things play out a little more before committing the time to deciding things. (I think that's a bad idea; regardless of the revolutionary tech, it seems wise to begin governmental thought early.)
It's kinda been buried in an otherwise heavy news cycle since the end of October of last year, AI and otherwise. I assume that wasn't intentional, but it seems hard to hide something so big at a better time.
And though I hate to say it, I suspect: apathy. Both traditional from the bottom ("too far off, unfixable, can't do anything about it"), and from the top ("once we get too big to fuck with, we just won't give a damn about any changes they try to make anyway").
Whenever government proposes banning a technology, one must ask: who benefits? The LLMs, even in their current state of infancy, are powerful tools. Restricting access to those tools keeps power in the hands of wealthy corporations and the government itself. That's the power aspect.
The money aspect is even simpler: Don't doubt that some of those wealthy corporations are making donations to certain officials, in order to gain support for actions like this. Almost no one leaves the Congress (or almost any parliamentary body in any country) as less than a multi-millionaire. Funny, how that works...
In the case of these models, you can fine-tune the model all you want to be moral and not do harmful things like scam the elderly, carry out disinformation campaigns, harass people to the point of suicide. But as soon as you release the model weights, you are giving anyone the ability to fine-tune out all of those restrictions, with orders of magnitude less cost that it took to develop the model in the first place.
Regulating AI, especially as it becomes AGI and beyond, is going to be very tricky, and if everyone has the ability to create their own un-restricted, potentially sociopathic intelligences by tweaking the safe models created under careful conditions by big labs, we're in for a lot of trouble. That assumes we put the proper regulations on the big labs, and that they have the ability to make them "safe", which is hard, yes. But as AI turns into AGI and beyond, things are going to go pretty nuts, so it's important to start laying groundwork now.
OpenAI's comment to the NTIA on open model weights - https://news.ycombinator.com/item?id=39900197 - April 2024 (41 comments)
It still sends the message, and if many people send such letters, it will force them to reconsider.
It doesn't. Pedophiles aren't high up on the priorities of the Federalist Society or Heritage Foundation or whoever Thomas' latest sugar daddy is.
They're going to flip the second one of their patrons is involved, regardless of precedent.
It's not that I think the US supreme court will never rule correctly in favor of speech or other basic freedoms. I'm sure they will, as long as none of their corrupt interests are threatened. As they say, "the rules only matter when the outcome doesn't".
The supreme court is certainly biased in favor of conservatives.
But conservatives these days are generally much more in favor of free speech and libertarian arguments than the people on the left.
So, for this specific issue, you should be happy that the supreme court is more conservative and willing to support free speech arguments.
That's totally wrong, it sounds almost like downplaying or excusing corruption (which is inherently a "bad thing" on its own) by comparing it to just having a bias.
Corruption is not just having a bias (towards "people who give me bribes", if nothing else) but also acting upon that bias and mixing it up with a job that requires impartiality.
In contrast, merely having bias is nowhere near as bad, particularly since someone can be strongly biased and still recuse themselves.
When someone talks about "corruption" on the Supreme Court, it's probably not hyperbole about bias, but a reference to hundreds of thousands of dollars in alleged bribes.
The people supporting open models are a much larger group than the minority who are trying to ban then.
Even without the scandals of gifts to SC judges, there's much room for judges to be corrupt.
I hope you are right, but I think there are some nuances here you aren't considering.
Courts have ruled that code is protected under the 1st Amendment because it is something created by human beings, and expresses the ideas and style of its human authors. There is a clear analogy to literary works – which is also supported by the precedent of copyright law protecting computer source code on the grounds that it is a type of literary work – and literary works are a core part of the 1st Amendment's scope as traditionally understood (even back to its original framers).
Whereas, model weights are just a bunch of numbers produced by an automated process. The legal argument that they should be protected by the 1st Amendment is much less clearcut. I would be happy if they were found to be so protected, but one ought to be careful to distinguish what one would like the law to be, from what it actually is.
It's generally really easy to mix one category of information with another. The DeCSS song is a great example of somebody using a little bit of cleverness to inextricably combine functional code with core protected political expression. Even if you thought "pure" code was not speech, removing the code from the song would destroy the message of the parts that clearly are speech, and there's no obvious alternative way to send that message. https://www.youtube.com/watch?v=PmLpLdGzNpo
... but you're right that it's not a slam dunk in real court. What is almost certain is that the argument's strong enough for any regulations to end up stayed while the issue worked its way through a very long legal process. Probably long enough to make the whole question moot.
Basically it's a waste of time for any US Government agency to try this.
Keep in mind the outcome of the DeCSS case - the Second Circuit did hold the DeCSS code to be protected speech, but it also upheld the DMCA as a constitutional restriction on that speech under intermediate scrutiny. The losing side decided not to appeal, because of the risk SCOTUS might uphold the decision
One could fine-tune an LLM to defend a particular political viewpoint - and you might thereby argue that model weights are expressive and hence within the scope of 1A.
But, if all the government does, is ban open model distribution above a particular parameter size, the government will argue that is a content-neutral restriction (it applies to all LLMs regardless of their viewpoint). It would essentially be a “time, place and manner” restriction, and thus subject to “intermediate scrutiny” - which requires it to be “an important government interest and substantially related to that interest”. And I’m afraid the government has a decent chance of winning there - they just need to convince the Court that AI risks are real, and this regulation is substantially related to controlling them - and I doubt they’ll struggle to convince SCOTUS justices of that (none of whom really understand AI), and if experts are telling them it is risky, they’ll believe them, and they will hesitate to stop the government from trying to regulate those risks
Whereas, if the US government went beyond a mere parameter size limit, and tried to impose “safety standards” on LLMs based on the kind of content they generate (disinformation, extremism, hate speech, etc) - that is harder to justify as content-neutral, and lack of content neutrality makes strict scrutiny apply, and the government is much more likely to lose there. The conservative justices especially are likely to view many of those standards as having a progressive political bias
> What is almost certain is that the argument's strong enough for any regulations to end up stayed while the issue worked its way through a very long legal process. Probably long enough to make the whole question moot.
Yes, it is likely to get tied up in the courts for a long time, and decent (but not certain) odds the regulations get suspended while the case proceeds, and the longer they are suspended for without the sky falling, it does tend to undermine the government’s case that they are substantially necessary. However, I’m not sure if that will really be decisive, because AI safety experts will always claim “we got lucky with this generation, the risk for next year’s or next decade’s remains very real”, and SCOTUS justices will probably believe them
They’ve also allowed non-content-based restrictions of that speech under intermediate scrutiny, and content-based restrictions under strict scrutiny, so that LLMs might be within the scope of things to which 1A protection applies does not mean that restrictions on distributing open LLM weights would necessarily violate the 1A.
I personally think the odds are decent that content-neutral parameter size-based distribution restrictions will be upheld under intermediate scrutiny. Personally I am opposed to such restrictions and think they are a stupid mistake, but I can separate my own feelings about a topic from how SCOTUS is likely to perceive it
Whereas, I’m sceptical that any content-based regulations (require LLMs to have safeguards to prevent them from generating disinformation, extremism, hate speech, etc) will survive strict scrutiny. Even if there is some category of content which is so horrible that SCOTUS will conclude the government is justified in restricting LLMs from producing it, it is basically technically impossible to design safeguards that only apply to that horrible content without collateral damage on less horrible content. Given that reality, I doubt any content-based LLM restrictions will be upheld
IMO free speech is permitting the broadcast of clear political thought and the freedom from being persecuted by government for expressing arbitrary political or religious ideas.
Enabling every person to have the information equivalent of a nuclear bomb doesn't seem like a first amendment issue. But I'm not a lawyer. Maybe I just don't like the first amendment anymore.
AI is like everyone having a Star Trek holodeck, but everyone can force anyone else into their holodeck and fool them with the most unethical or criminal scams.
People never grow tired of the ever-repeating excuses to further reduce our freedoms, or to hide criminality, or just insult the already lacking collective IQ.
Think of the children! We need to prevent abuse! Someone's feelings might be hurt! Someone might get harmed! Seeing real breasts might cause trauma!
Or one that's not so often used, but still really effective:
We're just bulldozering this place, because it's so horrible. Instead we turn it into a luxury resort for rich people. People who believe that it's because we're destroy any and all evidence are just conspiracy theorists. (in regards to Epstein's Island)
It should be keep in English culture. And its close friend allies. In my opinion, in our society, full of threats, it is not a good idea to make it open-source. (Not an expert on the topic).
That could allow adversaries parties, to obtain it. Parties that otherwise, would be unable to do so, by themselves.
In my opinion, it should be keep under control of the most able, and responsible organizations. Supervised by government. Because who else could supervise it?. Regulation of this tech, is a very important topic. That should be tackled by the best organizations, university and responsible researchers.
I would prefer a Star-Wars type of society. Where humans still do human activities.
I remain open to having my mind changed on this matter, but thus far, I’ve not seen a single good argument for restricting development of AI.
Personally I struggle to understand anyone who believes it's in humanities best interest to continue developing AI systems without significant limitations on the rate of progress. There's so many ways AI could go wrong I'd argue it's almost guaranteed that something will go wrong if we continue on this trajectory.
What's your opinion on the orthogonality thesis, ie intelligence not being fundamentally correlated with altruism? The concept of mesa-optimization, that a process that tries to teach a system to optimize for X might instead instead make it optimize for thing-that-leads-to-X (eg evolution made us want sex because it leads to reproduction, but now we have a lot of sex with birth control). The concept of instrumental convergence, that any optimization process powerful enough will eventually optimize for "develop agency, survive longer, accumulate more power"?
There's a lot of literature written about AI safety, a lot of it from very skeptical people, but it's not like there's any single insight that can be reliably transmitted to a guy at a table with a "you're all idiots, change my mind" sign.
I don't think were anywhere close to intelligence
>a process that tries to teach a system to optimize for X might instead instead make it optimize for thing-that-leads-to-X
Just another software bug to fix.
>any optimization process powerful enough will eventually optimize for "develop agency, survive longer, accumulate more power"
Pascal's wager for nerds.
>There's a lot of literature written about AI safety
There's also a lot of science fiction about killer robots exterminating humanity, same genre of writing.
You're mixing up your pre-written dismissals.
Instrumental convergence has nothing to do with Pascal's Wager. Pascal's Wager for nerds would be a somewhat valid answer to "AI has the potential to destroy the world therefore we should worry about it even if it's very unlikely", but that's not the argument I made.
A subtle, but important distinction.
Now, I keep getting stuck on this.
All of these models automate the creation of BS -- that is, stuff that kind of seems like it could be real but isn't.
I have no doubt there is significant economic value in this. But the world was awash in BS before these LLMs so we're really talking about a more cost effective way to saturate the sponge.
Anyway, on the main topic... closing models is an absurd idea, and one that cannot possibly work. I think the people who have billions at stake in these models are panicking, realizing the precarious and temporary nature of their lead in LLMs and are desperately trying to protect it. All that money bought them a technological lead that is evaporating a lot faster than they can figure out how to build a business model on it.
...Nvidia should pump a little bit of their windfall profits into counter lobbying since they have the most to gain/lose from open/closed models.
And no doubt a burgeoning resistance focused on making open source models available. It would be a disaster but I can see it making the industry stronger, with more variety and breaking the influence of some of the bigger players.
Practically, if you're concerned about this, learn more about ML/AI, not about how to use super high level frameworks but about how it actually works so when "SHTF" as survivalists say, you'll still be able to use it.
Because if this had happened in any non Western country, the mainstream news in US and EU would've been of the similar sentiment.
They don't control China or Europe, and will hand their overseas monopoly to overseas competition.
...but there are plenty of useful things that we ban the general public from having access to.
Opium poppies. Automatic weapons. Gas centrifuges.
In fact, if you start publishing detailed designs for the last one you're likely to have people in suits who only go by first names visiting you in pretty short order.
Yeah, war is bad. Sometimes the alternatives are worse.
It's more like trying to ban web or emails in one country because somebody can create harmful website or send harmful emails.
Sooner or later we will have to look at information through AI lenses. Before seeing anything it'll go through filtering and transformation to weed out spam, misinformation and do fact checks etc.
You can't build this defense future with crippled law.
Why is this statement assumed to be true? It is far from clear that advanced weapons in the hands of irresponsible, impulsive and ideological individuals cannot cause large scale chaos.
To build an egg requires the effort on the level of states or corporations but to break it requires just an individual motivated to do so.
With closed models and research only well funded entities participate (state actors and large corps).
If you want to draw parallels with advanced weapons, the difference is that weapon is also an antidote to itself.
Compare nukes. You can't stop a bad nuke with a good nuke. If I give you a nuke, the probability that you get nuked only goes up.
Afaict, nobody has even offered any explanation how open weights do anything to defend you from bad actors with LLMs.
So far, that's exactly what we've done. MAD works. Ask the Ukrainians -- or what's left of them -- what happens when you voluntarily disarm.
There maybe good reasons to have open models but pretending that systems that allow an individual to project power beyond their immediate vicinity is harmless is naive.
An individual with a weapon can almost always cause carnage at a level greater than their own with before any first responder can deal with it. Be that a viral video of someone faked to be blaspheming or somebody with a weapon in a school. Once the shooting starts it takes phenomenal energy to put the problem back in the box.
I'm just saying I'm open to it, and don't want them to listen to accelerationists, but rather the ppl doing the deepest work with the edge models. Many of them are humbled and worried. Generalist AI enthusiasts wanting freedom to do any and all things with paradigm-shifting intelligence infra, I'm not swayed by that so much.
The more I use LLMs the more I’m convinced they’re harmless… The world’s not going to end because people at the margins are asking LLMs for recipes to manufacture cocaine… or a nuclear bomb.
I don’t know that that idea always holds. But it definitely would for AI
All you need is ordinary commodity computing power to keep moving Ai forward. So vast numbers of individuals, organizations, and governmental actors will. Visibly or invisibly
Any blunt reactionary prohibition will only tilt things in favor of an unconstrained underground
>Many of them are humbled and worried
Yes, because most of them like to imagine themselves as the next Oppenheimer rather than accepting that they've invented a lossy compressed version of Wikipedia. Chatbots aren't nuclear bombs or "paradigm shifting intelligence infra". I wonder how long it'll take until the hysteria actually dies off.
It reminds me of a conversation with a Palantir guy who at least admitted that he likes it when the press writes a story that makes them look like a Bond villain because every time the stock price goes up. That's why AI luminaries are "worried".
This is one reason why I want AI to be as free and accessible as possible as quickly as possible. People are going to assume that AI is nothing short of magic until they get a chance to play around with it themselves. The sooner people stop panicking about things AI currently isn't and likely won't be for a very long time (if ever), the better we can focus on preventing it from being used inappropriately.
I don't think AI is going kill me in my sleep, or make us all unemployable, or brainwash me, but terrible AI could easily screw me over in all kinds of ways as companies and governments increasingly use it without oversight. AI replacing front line customer service/support is bad enough but police, doctors, and HR departments will happily use it and mistakes humans would have prevented will be inevitable.
But banning open models won't prevent any of that, it will just make the effects worse and leave people even more helpless to understand or tackle them. And it will remove much of the actual utility AI can provide individuals leaving us with nothing but braindead corpospeak models that refuse to tell you how to sue a corporation and won't even let you generate anything might run afoul of a large entities copyright. Literally worse than doing nothing would be consolidating this mess all under lobotomized corporate control.
Which impacts are you referring to?
Furthermore, it means that these entities can listen to everything being sent into the models.
If the model weights and training data are not published you can't even try to determine whether the model is designed to manipulate you, because you don't even have a fixed object to look at. If you find something weird about the model and start probing, they can just change the model to be less obvious.
This kind of thing: government or commercial manipulation of models is one of the main problems, and the ability to probe, fine-tune and and modify the models are what allows ordinary people to make them do what they want; and this is a free speech issue. LLMs are going to be critical for journalism, to sift through data, to sift through other people's news and characterising it objectively, etc., and that requires absolute control of the model, which makes access to the weights absolutely necessary.
If there are restrictions on LLMs, then OpenAI, the diverse governments, the CIA, etc. must be subject to as severe restrictions as I am.
Yes, it would be a disaster. That's the whole point. I think the field should be hit with a disaster, because it's broadly harmful for the species.
The abilities of LLMs are limited and there is reason to believe that this approach will hit a wall at some point where new strategies needs to be considered.
That said, luckily I believe it hard to put a lid on open weights again. A lid that would not solve any tangible problem anyway.