I am not cynical enough to believe that Anthropic's warnings are pure marketing hype. Let's hope that it is instead overconfidence or the result of too much time talking to their own chatbot.
Nor am I. I think they believe that AI poses a grave danger, and they are playing the prisoner's dilemma as an unvirtuous actor.
1. If anyone builds strong AI, it may be catastrophically bad.
2. If anyone builds strong AI, it will be better for the builder than for anyone who does not. Either because it won't be catastrophically bad so the builder will get to enjoy all the spoils indefinitely or because it will and at least the builder will be rich for a while.
Maybe they mean the AI needs to be safe from us? Can't have the grubby meat flappers touching the delicate bits!
This means their strategy is more like:
1. If someone builds a market-leading unsafe strong AI, it may be misused in a damaging way by a large number of humans, undermining society and creating a catastrophic upheaval.
2. However, if the leading AI maker also works to make it safe against misuse, as long as the stay in the lead and keep it safe, then the ability of human bad actors to misuse the AI is limited. Given enough time, society will adapt to pretty much anything, so eventually there's no longer an arms race to stay ahead.
I don't really know whether I agree with their concerns, but I do think that (my understanding of) their principles is that they're reasonable, self-consistent, and they adhere to them in all their public and private actions.
Some of us remember the same stories circulating in the late 90s -- where in a lab in Japan, someone had built a robot so advanced that it tried to escape from the factory. Which of course comes straight from 1960s science fiction.
The modern version of that now is Anthropic saying its AI can jailbreak itself out of its sandbox, etc etc.
Cynicism with these companies is highly warranted though. It's not doomerism to look at their actions and conclude they're deeply untrustworthy.
Sure there is. Intelligence doesn't give us our selfish motivations, natural selection does. We have similar motivations to C elegans, that has all of 302 neurons. Stay alive and have sex.
Honeybees don't though. They are about halfway between humans and C elegans when it comes to cognitive power. But they are not selfish because they don't reproduce directly (I'm talking about the worker bees). So they will sting even though it kills them. All their behavior is consistant with this.
I've had the same perspective for quite a while now, but hadn't been able to phrase it this cleverly.
Our neocortex is, by any definition, vastly more "intelligent" than the rest of our brain. Yet it doesn't attack the cerebellum. In fact, it takes orders from the older "lizard brain"!
It's not cynicism if it's an appraisal of reality that's backed up by evidence.
Remember how social media - that first baby of this current generation of tech entrepreneurs - was supposed to "bring the world together" and "let us express ourselves"? As it turns out there's a lot more money to be made by fostering division to drive engagement and feeding people an endless stream of ads instead of their friends' content. And money is what matters. You can't write down good vibes on a quarterly figures report. You can absolutely write down the number of eyes that your ragebait brought to a product's marketing efforts and the conversion rate to sales.
The same will be done with GenAI. We're being promised "AI Safety" because otherwise this whole thing gets killed dead by anyone who knows about James Cameron's directing career. There's no real enforcement mechanism for AI safety, though. Safety is a good vibe, same as harmony in online communities. You can't measure it. What you can measure is training costs and the cost of mistakes by AI that need to be trained to avoid those mistakes. Since AI generates more output than humans can conceivably QA no matter what your budget is, and since AI is seen by the market as a potential endless font of value, the tradeoff will be made to have AI make some potentially awful decisions while training itself over slowing down and re-appraising what is being done.
There's an almost religious reverence for AI in SV. Not everyone sees it as "making the godhead" but some certainly do. They're not going to moderate themselves too much on this.
I expect that Anthropic will eventually behave as you describe, like any other public corporation. However, my impression is that its current leaders are still more sincere than greedy.
Remember how OpenAI was supposed to make open-source models and cap its potential returns to investors at some multiple of their principal (my memory says 100x, maybe I'm wrong)? Well, that went out the window as soon as the word "trillion" was mentioned.
Whether you agree with that argument is another question.
Actions speak louder than words. If you want to understand someone, simply watch what they do. What they say is irrelevant.
So either they lie or they are AI Zealots. Interesting times.
> If nukes were not invented yet, would it really be a good idea to build and sell them as fast as possible (in peace time, no less)?
Arguably, yes.
From Richard Rhode's "The Making of the Atomic Bomb", I got the impression that most scientists involved thought they could manage a US or UN monopoly on nukes after the war. General Groves attempted to buy up all of the world's uranium ore. Unfortunately, it is only high grade ore that is rare; many countries have low-grade ore.
If only the US or UN had nukes we would't have MAD. We mostly got here through espionage
Who’s invading North Korea? No-one.
If in the WW2 Japan also had nukes (and delivery systems for them) they'd probably have retaliated in kind and US wouldn't let that slide too and it would have continued for some time.
This is a maybe. What we’ve seen so far, no two nuclear superpowers ever nuked each other, as they know both will suffer.
It doesn't really have to be dishonest, he could really believe it. I do believe, however, that it is incredibly wrong and is functioning as marketing hype.
So either they lie or they are AI Zealots. Interesting times.
Edit:
> > and the two people I knew who later joined Anthropic seem like the type to do it for the greater good instead of money.
There are three types of people. Pedestrians, investors, and “I know some of them, they wouldn’t lie”.