Most technologies aren't races
astralcodexten.substack.com
astralcodexten.substack.com
That's the weak point of this article.
Because the article then goes on to admit:
> The most consequential “races” have been for specific military technologies during wars; most famously, the US won the “race” for nuclear weapons. America’s enemies got nukes soon afterwards, but the brief moment of dominance was enough to win World War II.
And basically the fear is that AI is going to be exactly like nuclear weapons in terms of delivering an overwhelmingly decisive advantage, and that this could very much be concrete reality specifically in a war where China invades Taiwan.
In everything from integrating millions of information sources to determine best strategy and tactics, to controlling swarms of smart drones and bombs and missiles.
If there were ever a moment to compare something to a technological race like the nuclear bomb, I'd say AI fits it to a tee.
What would have looked different would be the political outcome. If USSR moved its armies to the East to fight Japan, well, entire Korea and half of Japan could have become Communist. On the other hand, with the Soviet Army engaged in the Far East, countries like Poland, Hungary and Czechoslovakia could have plausibly wrestled themselves free from the Soviet grip.
However, as it turns out, by the end of the war, Germany was actually pretty far from even seeing the finish line. For some reason, murdering or driving away many of your brightest minds because they happen to come from the wrong branch of the religious tree/diverge from the mainstream sexual orientation/don't adhere to the political cult in power doesn't help come up with breakthroughs in fundamental sciences.
And lots of people in the unaligned* world is ready to switch from San Fransisco to Beijing if the latter provides cheap LLM for commercial purposes.
* In the human sense of the world
Russia saw the writing on the wall and wanted to claim a piece of the pie before Japan's inevitable surrender. The only way it hastened the end of the war was in convincing the Japanese leadership to hurry up and sign the papers before any other countries declared war and made claims.
I didn't make that claim. But the USSR at the end of WWII was a force to reckon with anyway. They were certainly able to invade Sakhalin (and they indeed took the southern half of it) and destroy Japanese armies on the mainland.
I don't know enough about contemporary Soviet naval ability in order to judge if they were able to "island hop" from Sakhalin to Hokkaido and Hokkaido to Honshu. The stretches of water between the islands aren't very challenging in summer, but Hokkaido was very undeveloped at that time, so it is possible that lack of roads would stop the advance more efficiently than the sea.
It's a decision tree supporting the argument for a pause on AI research.
If it's a normal transformative technology, then a pause won't hurt because it's not a race.
If it is not, then if it is non-aligned it has huge potential for catastrophe. Therefore a pause is essential to ensure that we end up with an aligned AI rather than a non-aligned one.
Choice A: pause won't hurt. Choice B: pause prevents destruction of the world.
I'll take Choice C: alignment is a largely hypothetical problem that probably won't be a major issue at all, while an authoritarian China gaining an insurmountable military advantage would be a here-and-now disaster for those of us who value democracy and self-determination globally.
So I'm more concerned about a worryingly realistic problem (China gaining military dominance over the West due to winning an AI race) over a totally hypothetical idea that we're going to reach some kind of singularity where non-aligned AI destroys humanity.
If we start getting to a point where a non-aligned AI singularity seems to be a year or two away (or even a couple of months) then yes, shut it all down -- the same way you'd shut down a nuke that turned out to be able to obliterate the entire planet. But right now the singularity is entirely in the realm of science fiction and futurist philosophers.
So right now, yes AI does look like a race, and it's important that democracy win that race rather than authoritarianism.
I don't expect AGI at all. I expect very powerful domain-specific AI (e.g. integrating information, battlespace simulation, weapons coordination) that is far more effective than human coordination and communication can be, but I expect it to still just be sticking to pattern recognition, simulating scenarios, and picking the most likely scenarios based on relatively simple reward functions, and with constant human oversight (in the same way managers oversee their employees).
Today's chatbots are controlled perfectly in the sense that they're performing exactly how they were designed to. They're predicting a plausible next word, that's it.
If we start seeing rapidly progressing AGI then it's an entirely different conversation. But there's zero indication that's coming even in a couple of decades. It's 100% hypothetical right now.
For a Chinese military supremacy scenario, it would be hook the AIs up to all sorts of design, manufacturing, hacking and information warfare processes. Likely this would also secretly include biological, chemical and nanotech lab research as well. That sounds a little scary. It could even just be a stealthy/offensive enough missile design that could justify a first strike.
And there's no reason other capable governments won't do a similar thing if we're entering a new arms race. Will it matter so much when a democratic nation achieves supremacy, if the AI is busy designing devastating weapons in a semi-automatic, recursive fashion where nobody really understands the full potential? We're fortunate so far that nothing has triggered a nuclear war.
Think, just for one second, about how many technologies this applies to.
The internal combustion engine? massively powerful, easy to control. The electric grid? World changing, started as a huge mess of wires but is now, pretty much everywhere, well regulated. The internet? Absurdly huge, it should be as difficult to manage as an AI with the aggregate intelligence of millions of people, many of which are experts and bad actors. Despite this it is now essential and is under control enough that its advantages massively outweigh its disadvantages.
Any country that lacks any one of these three technologies is not a player on the world stage.
Tesla didn't "win" the EV market (and will likely dwindle), but it did become the most valuable car company and made Musk the richest person in the world (for a time).
OpenAI likely won't win the AI market, but they're landing multi-million deals left right and center. They've got the home advantage as well - US and EU companies don't want to deal with Chinese hard- or software.
It's not so much about winning, it's about getting a head start.
The iphone was in 2007. The Blackberry preceded it by many years. Same for the Palm Pilot.
Many traditional phone makers ceased to exist in the iPhone era — Blackberry, Nokia, Palm, Motorola, Ericsson, etc. Many new players took their places — Google, Samsung, Xiaomi, HTC. All smartphones are similar today. In that context Apple is a pioneer, with a head start. Wasn’t for long, mind you.
[0]-https://www.cultofmac.com/wp-content/uploads/2012/02/beforea...
If you include pre-iphone smartphones and pre-tesla EVs, you'll get the opposite conclusion: Apple and Tesla demonstrated that a head start isn't necessary to surpass competitors and achieve market dominance. By making better technology you can leapfrog companies (like Nokia) that have been around for decades.
So this is evidence that the first move is not always the winning stroke.
Apple won because of superior design and engineering. It became glaringly obvious when BlackBerry released the Storm one month or so before or after Apple released the iPhone 3G (and the App Store).
I remember thinking it was a prototype when someone showed a Storm to me. Scrolling was janky, the whole phone was sluggish and slow and the keyboard barely worked. Blackberry couldn't even figure out how to get Wifi on the thing, so it was stuck using cellular while my iPhone was just streaming video. And of course, no apps. It was 2 years behind the original iPhone and shipped two years later.
How much did this cost BlackBerry?
Edit: I did a quick Google Search and she was buying them on Ebay. I think it was one of the many misplays by the company.
Sure the iPhone introduced lots of cool new stuff. But some people on hacker news seem to think it happened in a vacuum and nobody else had ideas that mattered.
[1]: https://www.androidpolice.com/2020/06/27/features-apple-borr...
I'm not an expert on what happened to Dresden but I heard it was ... a lot.
Modern military doctrine is hitting a whole city is bad. You want to hit specific targets and not collateral damage to other things. Discredited WWII doctrine was killing civilians would make them surrender so hitting the whole city was good. Which lens you look at military doctrine says which is better.
The destruction caused by a single military operation was in the same league. The logistics were actually harder in some ways than traditional bombing. It would be months before the US would have more atomic weapons after the first two. So the one bomb versus hundreds doesn’t make a big difference when you have zero.
This would all change dramatically over the coming years. But at the time of its first use, the atomic bomb was not as much of a leap forward in capability as you’d probably think.
Unfortunately, public intellectuals in our current society don't hold themselves to these kinds of standards (which would have been considered obvious for public intellectuals a couple of centuries ago), and no one else holds them to it either.
Have you read anything a couple of centuries old? The problem was, if anything, much worse!
Dresden, Tokyo, Hanoi—how many more counterexamples do you need? The means to level a city with conventional bombs already existed. The nuclear bomb's innovation was as a weapon of intimidation, not raw destructive force.
Conventional strategic bombers and high explosives would never have given us MAD. The raw destructive force is absolutely relevant, it was (and remains) infeasible to do anything like wipe out a whole country in a day or two with conventional weapons, that'd take months or years and enormous dedication of resources. With nukes? Now it's possible.
I'm really not getting the reading of this as "LOL we could blow up cities with regular bombs, he's wrong!" Like, WTF was all the worry over nukes in the cold war, then? They're extremely obviously a major game-changer. A functioning nuke program means multiplying the max plausible destructive power a country can deploy inside a week by a very large number—no later developments in conventional weapons have really closed that gap, it's a whole other category of weapon, and you either have it, or you don't.
> there is no slightly-worse-nuke that can only destroy half a city.
Is itself false. The "Little Boy" that dropped on Hiroshima was the "slightly-worse-nuke" with a moderate blast damage radius (5 psi) of 1.67 km. The Dong Feng 5, China's largest nuclear warhead currently deployed, has a moderate blast damage radius (5 psi) of 24.5 km.
You still need to do every bit as much science and engineering to make it; the only difference is you need slightly less fissile material.
Nuclear powers can credibly threaten to not just destroy a single city, but to wipe a modern nation off the map. That's something you can't do with conventional bombers, even a lot of them.
I think his specific claim is not quite right here, but the thrust of his argument is. There's a reason that nuclear powers dare not enter direct conflict with each other in a way that major military powers used to casually do.
I think the key insight here is rate-of-production (measured in, say, tons of TNT equivalent, ready for delivery to a target), ease of stockpiling huge levels of destructive power, and cost of delivering that destructive power. The first two are incredibly higher for nukes vs. conventional weapons, and the last, far lower.
500 late-WWII Allied heavy bombers couldn't wipe out all the major cities in, say, the Soviet Union in a couple days. Impossible. The logistics simply don't work, there's no halfway-plausible scenario in which they could even get close to achieving that, there are tons of obstacles standing in the way.
.. with a few hundred nukes, though? Now it's totally possible.
Seems like other replies are trying to harp on this to undermine the entire article, when they are just deliberately misinterpreting the point. Maybe no analogy is perfect?
But I do think the broader point of nukes being a "you have it, or you don't and can't even really approach the capability of those who do" sort of tech is accurate.
Concentrating the damage like that gives the enemy far less time to adapt or rebuild, and less capacity to do so—you can out-produce heavy damage that's spread out over a long period of time, you can adapt to it, you can shift tactics and resources around to reduce the rate of damage in the future, but such efforts are hopeless against nukes.
It wasn't especially in-play during WWII since even the US couldn't (yet) build the bombs fast enough for them to really matter that much versus what could be done with conventional weapons, but it did matter soon after as stockpiles began to grow and production rate increased. Nukes represented a huge multiplier in the amount of destructive power a state could realistically stockpile and deploy in short order.
In other words, the basic historical claim being made here, that the US's "brief period of dominance" in nukes won WW II, is false. Not only that, it's "schoolboy error" false: the sort of mistake nobody who claims any sort of knowledge about the subject they're writing about should make.
- Delay thoughtful regulation of the technology - if you can delay it long enough until the entity gets big enough to fund lobbying efforts, its much easier to reach steady state/too big to fail status
- Gain access to government funding - as a consultant, if there's no existential crisis of being forever stuck in second place (no one wants to be behind), its much harder to provide thought leadership seminars/get fully paid vacations (i.e. speaking events).
If I owned a fully-robotic factory that produced all the goods this country consumed, unemployment would be at 100%, but GDP would be the same (Because the amount of wealth produced would be the same as it is today.)
Capitalists aren't going to share their profit until we take it from them, either through taxes[1], or, ah, 'land reform'[2].
[1] The carrot.
[2] The stick.
Output always creates more work, due to second law of thermodynamics (entropy). So, the more output you have, the more work you need to fight entropy. May be the Robots can deal with entropy too, but we aren't there yet.
So, AI will increase need of workforce / labor
...You're gonna have to explain that one, because I don't see how those things are connected at all.
Lets use car manufacturing example. Yes North America and Germany became early winners, and eventually Japan and Korea have become part of the winning club. But many other countries are not winning here. There are countries that can only make subpar cars for the local market, or make no cars at all and must import them all.
So if they are winning, what exactly are these countries winning? They are winning money and influence. US and Japan sell more cars overseas, that's more money into their country and into the government's coffers.
And who can say that being an early computer & Internet center didn't privilege America? The majority of big Internet companies are American, and they're now collecting rent across the globe!
The whole point of the article is that either AI is not that big of a deal and like many other technologies (e.g. electricity, cars, ... ), or it's so revolutionary that's it's so hard to align that we're doomed.
Well, can't it be something else? Can't it be something very safe which gives a country a 200 year head start?
Well, there is a difference between the "developed" and "developing" world, but that doesn't seem to be due to access to a single specific technology...
Imagine the Germans nuking London in 1941. Imagine the chilling effect this might have on the US public's willingness to enter WWII?
To say something had no impact is very different than it could have a MAJOR impact. And nuclear weapons are clearly an example of this.
AI is much the same, though I think 70% of the shift occurred with unmanned vehicles. The ability to send something with no person in it, to fight and kill a person is a HUGE game changer. When that 'something' gets better, smaller, and smarter... the edge becomes all the more decisive.
Also the ability to take in much more data... Imagine having 100,000 sensors you want to understand reasonably well. You may not have enough trained humans. But an AI? Maybe... And maybe it can correlate the signals etc.
AI is a game changer, and those saying it is as powerful as the bomb may not be underselling it. Mainly because of the precision and deniability the end results may offer.
* being first doesn't guarantee success (otherwise we would all own Xerox laptops right now)
* hard research and development is always a race in terms of getting to a product before running out of money
Basic economic theory shows there is always a scarcity of resources - money, time, market, people. The faster you are in front of the pack, the more likely you will get the lions share of money, the more people buying your product, and the more people wanting to join your venture.
The secret is to never fool yourself that you have unlimited time or money. That pathology leads companies to burn cash and take too much time choosing the "perfect" over the "really good", never getting to market. That's where I believe most startups failed in the past few years, and we're seeing that now more clearly with the desert of VC investment. You can fool yourself, but you cannot pretend basic economic forces don't exist.
I think there isn't going to be a single dominant LLM. There will be multiple competing ones. It's difficult to see how companies will achieve lock in. Natural language is the most flexible interface. A front end app could easily switch from one LLM to a different LLM behind the scenes with very little effort. It's basically one API endpoint...
How intelligent can the LLMs get and still be useful to humans? There is a point beyond which intelligence won't have any use for us. Humans may have already reached this point in some industries. Intelligence above a certain threshold doesn't have much value.
Most people who think the AI "race" is important aren't supposing this. So I don't think you can just "suppose" this away in the second sentence.
One thing that is a little different with these AIs is the ability to improve programming.
I bet, GPT-4 can make it simpler for humans to build/deploy GPT-5. GPT-5, could do even more of the heavy lifting for GPT-6.
The pause would give him time to spin up a startup and attempt to compete.
"If Tesla builds an AI, I'm smart enough that I can keep it in check. If someone else builds an AI their CEO is probably not smart enough to do the same thing."
Elon has a long history of thinking he's smarter than everyone else and that even people who are experts in their field aren't as capable as he is (cave submarine).
https://www.cnbc.com/2017/09/04/elon-musk-says-global-race-f...
> “Last month he co-signed a letter with other leaders in robotics including Google Deep Mind’s Mustafa Suleyman, calling on the United Nations to ban killer robots”
How can this be true if AI is not a race? What does it even mean to be behind in a technology that isn't a race?
This isn't rocket science.