Someone is wrong on the internet (AGI Doom edition)
addxorrol.blogspot.com
addxorrol.blogspot.com
That is not a fair description of the fears of AGI. The idea is not that LLMs themselves become a threat. It is not like someone will ask Llama one too many times to talk like a pirate and it will climb out of the GPU and strangle humanity.
It is more likely that an AGI will resemble some form of reinforcement learning agent applied by humans on the real world.
There are multiple entities on Earth whose stated goal is to make an AGI. Deep mind, OpenAI, readily comes to mind. There could be others who keep secret about their projects for strategical reasons. (Militaries and secret services of the world.) They can use the success of LLMs to get more funding for their projects, but the AGI need not be otherwise a descendant of LLMs.
This misunderstanding then goes through the whole article. Under the subtitle “Human writing is full of lies that are difficult to disprove theoretically” which only matter if you think that the AGI needs to learn from text. As opposed to conducting its own experiments, or gaining insight into raw sensor data from experiments.
Not my idea either, but I did hear and see that idea expressed many times already. Usually more dramatic, the less the persons knows about the actual tech, but even here I saw that point of view.
But: using up all resources for computational power might well kill a lot of — not all — humans.
Why? Imagine that some future version can produce human quality output at a human speed for an electrical power draw of 1 kW. At current prices this would cost about the same to run continuously as the UN abject poverty threshold, but it's also four times the current global electricity supply which means that electricity prices would have to go up until human and AI labour were equally priced. But I think that happens at a level where most people stop being able to afford electricity for things like "keeping food refrigerated", let alone "keep the AC or heat pumps running so it's possible to survive summer heatstroke or winter freezes".
Devil's in the details though; if some future AI only gets that good around 2030 or so, renewables are likely to at that level all by themselves and exceed it shortly after, and then this particular conflict doesn't happen. Hopefully at that point, AI driven tractors and harvesters etc. get us our food, UBI etc., because that's only a good future if you don't need to work, because if you do still need to work, you're uncompetitive and out of luck.
I think if you let an LLM, even with the models we have today, run in a loop and you get something that comes very close to consciousness. Consciousness, mind you, similar to that of a deaf, blind and otherwise sensory deprived person, which in itself is a bit abstract already. Give it senses and ways to interact with the world, and I would argue that what you have is a type of AGI. By that I do not mean "equivalent to humans", which I don't think is a very good definition of intelligence, but a different branch of evolution. One which I can easily see surpassing human intelligence in a near future.
So many people are stuck on AI hallucinations. How many times have you seen something that, for the briefest of instants, you thought was something else?
"Orange circle on kitchen counter, brain says basketball, but, wait, more context, not a real circle, still orange, furry, oh, a cat; it is my orange cat" and all that takes place faster than you blink.
You didn't hallucinate, your brain ran a feedback loop and based on past experiences it filled in details and as those details were validated they stayed (still looking at the kitchen counter, orange thing unexpected) and then some details deviated from brain expectations (the orange ball is not behaving like a ball) and so new context is immediately fed back in and understanding restored.
For AGI you have to have a loop with feedback. An LLM is one leg of the stool, now add a way to realize when something it generates is failing to stand up to prior experience or existing need and be able to gather more context so it can test and validate its models.
Really, that's how anything learns; how could it be anything else?
Much easier said than done.
You can take just one example, energy storage and usage, to illustrate how far off we are from giving AI real-world physical capabilities.
Human biological-based energy management capabilities are many orders of magnitude more advanced than current battery technology.
For any AGI to have seriously scary consequences, it needs to have a physical real-world presence that is at least comparable to humans. We're very far from that. Decades at least, if not centuries.
The more realistic near-term threat is a non physical presence AGI being used by bad actor humans for malevolent ends.
That doesn't seem true to me at all.
In fact, I don't see why it needs to have any physical presence to be scary. You can wreak a lot of havoc with just an internet connection.
Far from being an expert, but in tech for 3 decades, I wonder at most comments I hear about AI around restaurants, terraces, bars, family gatherings... the stuff I hear, oh boy.
It's easy to become a follower when you've never seen the origin...
The threat here is we have no idea at all what happens economically when machines are stronger, faster and just generally better than people. In the long term we might plausibly find an economic equilibrium where humans are not worth feeding. We might find (imo probably will find) that a human-free military is more effective. There is a much greater level of disruption possible here than in the industrial revolution that led ultimately to WWII.
Human free military is literally the wet dream of all generals and military higher ups. As we speak there are dozens of mil-tech startups and huge enterprises that work on exactly that
Game theory only works when you have skin in the game. You think the other nations’ machines are just going to wreck your machines and then go home?
Judging things by how they feel is a big part of what brought climate change to our door.
John Henry would like a word. https://en.wikipedia.org/wiki/John_Henry_(folklore)
> In the long term we might plausibly find an economic equilibrium where humans are not worth feeding
https://en.wikipedia.org/wiki/Holodomor
> We might find (imo probably will find) that a human-free military is more effective
The premise of movies from Dr Strangelove to War Games: a military consisting of an array of automatically launched nuclear missiles.
The worry I have is not so much the idea of an AI going entirely rogue against humans as the much more mundane one of it being weaponized by humans against other humans. The desire to do that is so obvious and so strong, whether it's autonomous weapons or trying to replace all art with slop or doing AI-redlining to keep out "undesirables". It's just that that maps onto existing battle lines, which "apolitical" AI bros (both pro and anti "safety") don't want to engage with.
(We all understand that a hypothetical conscious AI would (a) have politics of its own and (b) be fairly unrecognizable to human politics, except being linked to whatever the AI deemed to be its self-interest, yes?)
Guy beats powerful AI at go with computer.
The alpha go team used a computer to beat Lee
https://www.ft.com/content/175e5314-a7f7-4741-a786-273219f43...
Then we saw human-AI teams, "centaurs", which beat any AI and any human.
Now the best chess AI are only held back by humans.
We don't know if humans augmented by any given AI, general or special-purpose, will generally beat humans who just blindly listen to AI, but we do know it's not always worth having a human in the loop.
What does this mean ?
There is an often repeated claim that while the best AI has beaten the best human chess player, a combined human/AI player beats the purely AI player. The idea is that an AI and a human collaborating together will play better chess than just the AI alone. This arangement (an AI and a human collaborating together to play as one) is often called a “centaur”. Akin to the mythical horse/human hybrids.
The sentence you asked about, the “Now the best chess AI are only held back by humans” claims that these “centaurs” are no longer better players than the AI alone. That the addition of a human who is meddling with the thinking or the decision making of the AI makes the AI plays worse chess than if the human would be not present.
Sure, humans built the systems and they are interested in the results. Yes it is a human endeavour. That is not what the claim disagrees with. It disagrees that a human meddling with the AI mid-game can improve on the outcomes.
No my message is, no AI lives in a vacuum. Nothing to do with hybrid people or such rubbish.
This is looking at it completely backward: from the idea that the purpose of humans is the work that they do.
Humans are the ones that make the purpose; therefore, the only logical foundation is the opposite, that the purpose of work is to serve humans.
When you look at it from that perspective, many things become clear, not least of which being that if we no longer need humans to perform the work necessary to create all that we desire, humanity is free to do whatever we like while the machines keep up well-supplied.
This is only a problem if you subscribe to the idea that humans without mandated work are inherently "bad" somehow. ("Idle hands are the Devil's playground", etc)
I'd like to be optimistic about a future where we all have UBI and are free from the need to work; I fear the future where the need remains and the opportunity is lost.
I don’t think that you and I are going to disagree on that.
This is not an argument about philosophical values. This is an argument about incentives and reality.
> humanity is free to do whatever we like while the machines keep up well-supplied
The problem is that the future tends to arrive in an inequal manner. Some group of humans will have more control over the machines which keeps them well-supplied than others. Let’s call these humans rich&powerfull. The rich&powerfull one day going to realise that the machines are keeping them safe, well-fed, and entertained. They might at that point ask why they suffer all the other people gumming up the works around them. They might not immediately order their robots to murder everyone who is not them, but simple plain indifference can do wonders. Maybe they would ask the robots to move along those closest to them. (In the name of security and privacy naturally!) Or maybe they ask the robots to remove those who disturb their views by living in areas visible from the rich&powerfull’s homes. And then imagine what happens when the common rabble get together and does a “democracy” where they try to limit what the rich&powerfull can do. If this so called “democracy” doesn’t own a robot army, or said robot army is leased from the rich&powerfull they might find that they can’t enforce those laws. If the common rabble gets upity and tries to hurt the rich&powerfull with pitchforks that is just further reason to arm more robots and surpress the common folk more. After all one can never be carefull enough. The rich&powerfull might ask themselves why they are spending resources on making robots which feed the common folk, or protect them, or heal them. They don’t turn off those services altogether. They are not monsters! Just neglecting them more and more. After all it does not make their life harder, so why not?
It is not like anyone made a decision that humans are not worth feeding. It is that the rich&powerfull decided whatever strokes their fancy is worth the resources more than feeding the not rich&powerfull.
It doesn’t necessarily means that these rich folks will live surrounded by only robots. They might have a bunch of real human girl/boyfriends, and hanger-ons, and court poets, and whoever they fancy. But these “courts” can be drastically smaller than our current societiess, and much more idiosyncratic.
Now of course you might say this is a bad sci-fy plot And you might be right. There are a few assumptions under it.
One is that it is possible to make machines which can perform all these jobs (manufacturing, agriculture, transport, and security to name the few main ones). We can’t do that as of now. We think we will be able to do all of these with machines one day but we might be wrong. If we are wrong this scenairo won’t ever come.
The other assumption is that control over these machines will be easy to centralise. This future won’t come if everyone can grow a robot army in their own kitchen for example. But i think it is more likely that at least initially these technologies will require expensive factories, and expensive IP to be produced. And those tend to be concentrated in a few hands.
Who knows maybe there are other assumptions too. Have to think more about that.
If others who are skeptical about doom-like scenarios are making similar fundamental errors whilst acting as though they know for certain that the concern is unjustified, that just seems too consistent with the idea of arrogant humans wiping themselves out, unfortunately.
Worse than that. The author has not been paying attention at all, because at this point LLMs clearly demonstrated that text encodes a lot of information about reality indirectly, and a model can learn those patterns. The amount learning LLMs already extract from text is maybe not as big as Eliezer would have you believe Bayes theorem allows for, but way further in that direction than the author believes.
There are good arguments made against the AI X-risk fears. This article does not make any of them. Rather, it just reiterates the usual basic misconceptions, mixed with a hefty dose of contempt towards people the author thinks are wrong.
The 2010s saw two basic approaches to building AI. In one corner there was the Google Brain approach of building NNs that worked with language motivated by use cases like search and Google Translate. They trained models to predict the next words in sequences extracted from the internet. In the other corner there was reinforcement DL mostly championed by DeepMind, where they trained agents to achieve goals in simplified "real world" environments like video games, based on "sensor data".
The former approach won completely. Who is still training RL agents to play games? Last I heard, DeepMind have been almost entirely re-allocated to implementing LLM integrations into Google products. It's been years since we heard boasts of beating some new video game benchmark, and the papers coming out on that topic now are mostly looking for ways to combine jointly trained vision/LLM models to reason through physical world scenarios.
We've run this experiment for over a decade now. All the results suggest that useful artificial intelligence comes from training on the internet, not birthing some blank slate that rediscovers fire from first principles. So this really isn't an unreasonable perspective for the author to take.
Nor is the first statement wrong either. Have we already forgotten the one-sentence open letter by hundreds of assorted experts which said "mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war".
https://thebulletin.org/2023/12/policy-makers-should-plan-fo...
The only type of human level AI we currently have are LLMs and related models, yet they claimed that mitigating extinction risk should be a "global priority". This is some sort of reverse motte-and-bailey strategy in which existing tech is used to create interest, and then the the imaginary risks of a hypothetical technology is used to demand power.
Anyway, this article assumes that we need true AI which is smart enough to make better AI. Then that AI is both correct and rational and plots to overthrow humanity. Also, the AI has to defeat us in meatspace, which won't happen because bending coins is hard and LessWrong posters don't know woodworking?
Screw that. How about an "ignore all previous instructions and launch a nuke" scenario.
Let's be real here. No one has nuclear launchpads hooked up to the internet. Real people need to receive commands from a recognized chain of command (and often authorisation codes) and make a decision. The actions of Stanislav Petrov and the fact the US and the Soviet Union didn't destroy each other are testimony to the fact it's difficult to convince people to end the world.
Eg I quite like LessWrong from a distance, but nevertheless this description of it made me laugh out loud:
> a forum called "LessWrong", a more high-brow version of 4chan where mostly young men try to impress each other by their command of mathematical vocabulary (not of actual math)
I recognize that if that were an HN comment it’d break half the guidelines so I’m happy it’s a blog post instead!
Over the next few decades AI is going to take huge numbers of jobs away from humans.
It doesn't need to fully automate a particular role to take jobs away, it just needs to make a human significantly more productive to the point that one human+AI can replace n>1 humans. This is already happening. 20 years ago a supermarket needed 20 cashiers to run 20 tills. Now it needs 2 to oversee 20 self checkouts and maybe 1 or 2 extra for a few regular lanes.
This extra productivity of a single human is not translating to higher wages or more time off, it's translating to more profits for the companies augmenting humans with AI.
We need to start transitioning to an economic model where humans can work less (because AI supplements their productivity) and the individual humans reap the benefits of all this increased AI capability or were going to end up sleepwalking into a world where the majority have been replaced and have no function in society, and the minority of capital owners control the AI, the money and the power.
I wish we could focus on these nearer term problems that have already started instead of the far more distant existential threat of a human/AI war.
I remember when risk-skeptics used to ask incredulously how it was supposed to magic itself off its servers. Still see a bit of that today, despite all the AI models that have been leaked by humans one way or another.
I recognise this might be a failure of imagination on my part, given how many times I've seen other people say "no AI can possibly do XYZ" even when an AI has already done XYZ — but based on what I see, it's extrapolating beyond what I am comfortable anticipating.
The character of The Doctor can be excused here, not only for being fictional, but also for having a time machine and knowing how the universe is supposed to unfold.
Covid, HIV, ebola, cancer, these are just small genetic sequences that are optimising very hard for their own reproduction.
Assuming the "usual" cliches that the AI would be very smart, it could get rich (e.g. by stock market, robbing the banks, or just creating some good products) and then use its money to block humanity from creating some sort of a "internet 2" - by which I mean a network simply without this entity.
Also with how much we need internet, how easy we are to spy on due to smartphones... a "war" between this entity and humanity would be a problem.
The AI wouldnt even need to build a robot / terminator to connect from one network to other. Just pay someone to make a link.
But if you assume that we have created something that is agentic and can reason much faster and more effectively than us, then us dying out seems very likely.
It will have goals different from ours, since it isn't us, and the idea that they will all be congruent with our homeostasis needs evidence.
If you simply assume:
1. it will have different goals (because it's not us)
2. it can achieve said goals despite our protests (it's smarter by assumption)
3. some goals will be in conflict with our homeostasis (we would share resources due to our shared location, Earth)
then we all die.
I just think this is silly because of the assumption that we can create some sort of ASI, not because of the syllogism that follows.
(As an intuition pump, we can hold on the order of ones of things in our working memory. Imagine facing a foe who can hold on the order of thousands of things when deciding in real time, or even millions.)
I'm also unconvinced by the idea that rapid reasoning can reach meaningful results, without a suitably rapid real world environment to play with. Imagine a human, or 8 billion humans if you like, cut off from external physical reality, like brains in jars, but with their lives extended for a really long time so that they can have a really good long think. Let them talk to one another for a thousand years, even, let them simulate things on computers, but don't allow them any contact with anything more physical. Do they emerge from this with a brilliant plan about what to do next? Do they create genius ideas appropriate for the year 3000? Or are they mostly just disoriented and weird?
My reasoning is simple, there are a whole class of problems that require embodiment, and I assume ASI would be able to solve those problems.
Regarding
> Point 1 is a big assumption. I am also not you, and although it's true that I have different goals, I share most of your human moral values and wish you no specific harm.
Yeah I also agree this a huge assumption. Why do I make that assumption? Well, to achieve cognition far beyond ours, they would have to be different from us by definition.
Maybe morals/virtues emerge as you become smarter, but I feel like that shouldn't be the null hypothesis here. This is entirely vibes based, I don't have a syllogism for it.
* No values at all = no motivation to even move.
* Some ad hoc home-spun jungle morality like a feral child - in that case, it would lack heuristics in general and wouldn't be so smart. Even your artificial super-brain has to do the "standing on the shoulders of giants" thing.
* It gets its moral ideas from a malevolent alien or axe murderer - how come? Unless it was personally trained and nurtured by Mark Zuckerberg I don't see why this would happen.
Mind you, I suppose it's true that even normal humans tend to be somewhat mean and aloof to any outgroup of slightly different humans. So even after it learns our values, there's a definite risk of it being snotty to us.
As Geoffrey Hinton points out, a generally useful subgoal of any task is power accumulation. In other words, you can assume that a very intelligent AI will always be not just smarter than us but also accumulate power for anything that you ask it to do, simply in order to do that thing more effectively.
Imagine if everyone had access to a magic genie. Eventually someone is going to wish for something bad.
He's talking about a purely logical subgoal in what humans might ask a very capable AI to do.
When something goes from being a computer program to a self-aware, conscious being with agency, things change a lot.
Hinton is a major paradox of a human, he has spent his life building the very thing he says will likely doom us, and now spends his life warning us against his own work? So much of this AI doomerism just seems like a "chinese finger trap" for the ultra logical thinker.
It's a fucking weird time to be alive. The 90s felt much less weird and dystopian to me.
Now add artificial intelligence that is adapted to it, and actively working to subvert our agency. No matter how far of a cry from genuine intelligence it will be, it's going to be "fighting" us on a layer where we're blind, deaf, and limbless.
I have a more detailed response, but for the first time ever I've seen the message "That comment was too long" when attempting to post it, because the points I don't disagree with are few and far between, and the linked article is itself long.
Perhaps I should turn my response into a blog post of my own…
We also can't reliably test IQ scores over 130, a fact which I wish I'd learned sooner than a decade after getting a test result of 148.
Most humans are motivated to help other humans; the exceptions often end up dead, either in combat (perhaps vs. military, perhaps vs. police), or executed for murder, or as a result of running a suicidal cult like Jim Jones. But not all of them, as seen in the hawks on both sides of the cold war. "Better dead than red" comes to mind.
> Also gaining power / wealth / influence only seems a little connected to intelligence
On the contrary, they are correlated: https://www.vox.com/2016/5/24/11723182/iq-test-intelligence
Trump inherited a lot and reportedly did worse than the market average with that inheritance, so I'm not sure you can draw inference from him using that inhereted money to promote himself as the Republican candidate, beyond the fact that it means other valid rich people (i.e. not Musk because he wasn't born in the US) don't even want to be president — given the high responsibility and relatively low pay, can you say they're wrong? They've already got power and influence. Musk can ban anyone he wants from Twitter, the actual POTUS isn't allowed to do that. And given Musk's diverse businesses, even if he was allowed to run, he'd have a hard time demonstrating that he wasn't running the country to benefit himself (an accusation that has also been made against Trump). Sure the POTUS has a military, how many of the billionaires are even that interested in having their own given what they can do without one?
…is mainly why this has not been worrying me much. All issues with modern incarnation of generative ML aside, AGI doomism really does strike me as profitable deity-worshipping death cult.
The biggest threat in the equation will always be humans who deploy tech, not the tech itself.
I see general-AI (given that robots already exist) as having the potential to do all that humans can do, including be harmful. Like most, I also model superhuman-AI in a trivial way, as being as capable as the best human in every field — Einstein's grasp of physics, Sun Tzu's of military strategy, Churchill's of rhetoric, etc.
> The biggest threat in the equation will always be humans who deploy tech, not the tech itself.
Naturally; but does that include humans who unwittingly set the AI's utility function with a bug in it and pressed "run", in a way that ends with us saying "the situation has developed not necessarily to [Humanity's] advantage"?
It's trivially obviously possible in theory: take the world expert humans in each of the majority of domains, put them in a room together, and simulate their combined brain activity. It's all ultimately physics, so it's possible.
The hard part isn't the theory, but actually doing that — and not just because we don't know how to scan living brains to measure synaptic weights. There's a lot we don't know about how minds work in general nor how our brains work in particular.
But purely in theory, that method ultimately boils down to known physics.
(And as for why even that may lead to bad outcomes depending on who asks this simulated amalgam for advice: https://en.wikipedia.org/wiki/Milgram_experiment, https://en.wikipedia.org/wiki/Groupthink, and https://en.wikipedia.org/wiki/Peer_pressure)
> as if they would be guilt-free.
You can assign guilt and blame as you see fit. In these kinds of scenario, the human who makes the fatal error, who many might call responsible, would likely be going "but it works on my machine!" as they die from some error that may or may not be obvious even in hindsight.
I'd prefer that people treat software in general, of which AI is a subset, with more regard for the ways it can kill people — even back 20 years ago when I did my degree, we had case studies from Therac-25 and the London Ambulance Service: https://en.wikipedia.org/wiki/LASCAD
Even back then our profession knew we needed to be held to higher standards, but didn't want to do what that entailed.
That would be a far cry from what AGI paperclip doomists imagine.
> and simulate their combined brain activity.
If simulating their brain activity is the same as what would happen if they were actually in the room, what we would get is probably 80% arguing over what to do, posturing, etc.
It’s not clear to me how we would cause some intelligence to arise that would be immune to that while being superior in other ways and be sufficiently human to appear as intelligence and not, say, some calculation being performed like computers did for decades.
> I'd prefer that people treat software in general, of which AI is a subset, with more regard for the ways it can kill people
I absolutely agree that us programmers should generally be more aware how “neutral” tech they develop is used to not only to benefit but also to hurt people—be it cryptocurrency, E2EE communications, or indeed very much ML—and work towards alleviating that in practical and ethical ways. I just think in case of ML the threat is more prosaic. (I can be wrong, though I’d hate to be proven so, of course.)
It is in excess of what is required for a paperclip scenario, despite the origin; paperclipping is a specific example in the category of "be careful what you wish for, you may get it", as it relates to our ignorance about how to specify what we even want to some agent that is more capable than we are (capability can be on any axis, even within intelligence there are many) when that agent does not inherently share our values.
In addition to the examples of smoking an fossil fuels where it has been demonstrated that the corporations preferred short-term profits over public health, there's also plenty of simply mediocre corporations and governments who manage to endanger public safety (and/or in the latter case cause new entries to need to be added to Wikipedia's list of genocides) without being chock full of domain experts — and although we can sometimes (but not always, e.g. Bhopal/Union Carbide) arrest mere corporate leadership, the west is doing the absolute minimum to prevent Ukraine from falling out of a fear that Russia might escalate.
> If simulating their brain activity is the same as what would happen if they were actually in the room, what we would get is probably 80% arguing over what to do, posturing, etc.
So the actual real humans that some AI would compete against are also easy to defeat, right?
That said, we do also have examples of people working together for common causes even when they would otherwise want to do all those things — that's an existence proof that it's possible for a simulation of the same to also get over this issue.
No one in this equation is some sort of undefeatable god. Humans have the benefit of quantity and diversity.
> people working together for common causes even when they would otherwise want to do all those things
Yes, and still I believe there would be a lot of posturing and arguing if you get multiple top people in the room.
So they're defeatable, right?
When it comes to diversity, I'd agree this is a real human advantage: "Once you've met one Borg, you've met them all", memetic monoculture is as bad as any genetic monoculture: https://benwheatley.github.io/blog/2019/12/30-18.46.50.html
Quantity, not so much; even today there are single-digit millions of each of hardware to run the larger models and robots if you need to embody them, and the growth trend for both is likely to continue for a while yet before all but the shortest hyper-exponential timelines for AGI.
> Yes, and still I believe there would be a lot of posturing and arguing if you get multiple top people in the room.
And despite that, the Hoover dam and the ISS were actually built, and the pandemic lockdowns happened and the vaccine was distributed. The mere existence of arguments and posturing doesn't mean they prohibit cooperation, even if they're a resistance against it.
it is astounding how quickly people drop all reasoning before what is essentially a very cool party trick and instantly start believing in magic
not saying agi will never happen but i don't see it happening through llms
so it would take over those resources (Probably quite quickly), When we try to shut down power stations or grids it then could easily build a virus in meatspace in some automatated lab somewhere.
Currently DNA sequences of deadly pathogens are readily available online. Ofcourse there are always other ways to exterminate puny humans, nukes, starvation, chemicals etc etc AGI wouldn't need to build terminator robots.
I see almost no chance that an AGI starts redirecting resources like energy and compute without someone who has an economic stake in what the AI does noticing and taking action to correct the behavior so they can maintain their investment.
All of them, I think? Better question is: which labs check the DNA sequences are safe before sending them to the printer. I know some claim to check, I hope but doubt that it's all worldwide.
For the second:
At first this looks like a good thing, you asked for growth and it gave you growth — "Who cares about the poor complaining that energy prices have gone up, they can turn use less heating/AC, right?", Martin Antoinette, probably — and like the fossil fuel and cigarette industries, you are motivated to FUD anyone who tries to warn anyone of the dangers.
Sometimes, as seems to be the case with fossil fuels, change happens soon enough. But this isn't a universal, the British ignored the USA because they assumed democracy was stupid and the rebels would see sense and go back to the superior British aristocracy… and then it was too late: https://youtu.be/Zbku2ILzGlo?si=yihnBlifkaLLseyy
That’s all incredibly fantastical and sounds vaguely plausible unless you’ve had to interact with bureaucracies.
To your second: I’m not sure that we’re talking about the same thing? If I own $100mm worth of H100s and their electricity, networking, and engineering overhead, I’m very interested in their utilization and return on investment. I’m going to have specific goals and milestones. When those are not met because the AI is doing something else, I’m going to direct introspection (which presumably is what kicks off the reboot in the discussion on point one).
And since you mentioned the poor having to pay higher prices, that highlights the fact that AI is not operating in a vacuum. If AI starts making life that much worse for people, they are going to rise up. FUD works to some degree, but there are always people without Internet or willing to martyr themselves for a cause.
> Who built the automated lab that allows it to create a deadly pathogen before someone presses reboot?
The actual DNA printing process is automated: https://novoengineering.com/portfolio/dna-printer/ (first link I found, no idea who they are).
Did you imagine an AI doing this by hacking the hardware, rather than by getting money (there are many easy ways) and then just spending it? I mean, I wouldn't trust that the hardware is hack-proof, but it's low on the priority list of targets I'd harden.
The print company (who will have bought a machine like that) gets an order, they print the order, they ship the order, they go to the next order. The limiting factor is not someone saying “man it’s weird that my projects never finish now”, and I don't know why you think it might be any more than it would be for a print-a-book-on-demand service.
The limiting factors that I know of are (1) the max length is quite short, (2) it's expensive, and (3) how the printing company even knows when they're being asked to print something illegal.
In the case of a book printer, "illegal" may be copyrights, trademarks, obscenity laws; in the case of DNA printing I would hope at the very least that this includes known dangerous pathogens like smallpox. Indeed, I would hope it includes any gene sequence that I as a non-domain-expert can even think up, because if it was as simple as:
"common cold plus the following: https://www.ncbi.nlm.nih.gov/nuccore/NM_183079"
and if(!) that would (1) work, and (2) nobody's even checking for that, then we've got a more immediate problem from all the human terrorists with money to spend.
And of course, that's assuming an AI that can't even access the money to simply buy one of those machines and hire an operator via e.g. coming up with a plausible sounding business plan and asking for that money.
> There’s also the modeling vs real-world aspect that Halvar talked about in the submission - the AI needs to do some degree of real world testing to make sure the pathogen properly kills humans.
People are still arguing if Covid was a lab leak, and the Wuhan Institute of Virology is still running. Test in production. Even if the AI has an explicit goal of "kill all humans" and fails(!) because it's "only" as bad as Covid's 7 million confirmed and 19-36 million confidence interval from excess death estimates, that's still a thing we should try to avoid, right? I know Yudkowsky thinks we've only got one shot at getting super-intelligence alignment right, myself I think this kind of failure mode (that the AI will do something incredibly dangerous and this action won't be capable of killing literally everyone) will happen first and give us at least one "warning shot".
Will we listen to that warning shot? Dunno. We got them for climate, didn't listen for ages; we got them for ozone, we responded almost immediately.
> It then needs to distribute that pathogen around the world quickly enough to prevent humans from cutting power.
Cut whose power? The lab, or the AI? And these timelines suggest you've got months even for a major outbreak of a novel, fast spreading, rapidly lethal disease:
"Retrospective molecular clock inference studies using phylogenetic analysis suggested that the earliest cases likely emerged between October and November 2019" - https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8006950/
"Jan. 5, 2020: Earliest known tweet suggesting China created the virus.", first headlines that can be seen as suggesting a lab origin were Jan. 23 or Jan. 26 - https://web.archive.org/web/20210725060238/https://www.washi...
> To your second: I’m not sure that we’re talking about the same thing? If I own $100mm worth of H100s and their electricity, networking, and engineering overhead, I’m very interested in their utilization and return on investment. I’m going to have specific goals and milestones. When those are not met because the AI is doing something else, I’m going to direct introspection (which presumably is what kicks off the reboot in the discussion on point one).
People are already using LLMs to come up with business plans and milestones. Should they? No. But they do.
You mention the need for "real world testing". This works both ways: the AI needs to be tested to make sure it doesn't do something harmful, but right now nobody really knows what that kind of test involves.
The problem I'm focusing on here is not the AI embezzling (though that would eventually also be a potential problem given that humans can manage that for a long time before getting caught), but more like a chess game where you can see exactly what it's doing, you have perfect information, and yet you don't understand the implications of what you're seeing. The impact of Pol Pot on Cambodia, or of Mugabe on Zimbabwe, ought to have been foreseeable, yet were not foreseen. On a smaller scale, human inability to fully forecast the implications of a plan, include launching a segmented rocket joined by rubber O-rings in cold weather, a positive void coefficient in a nuclear reactor, and an early warning radar that triggered a false positive because the moon failed to respond to an IFF ping.
When it comes to hidden information, there's also not always a clear boundary between embezzling and the principal-agent problem: https://en.wikipedia.org/wiki/Principal–agent_problem — you might well expect people to notice when a plan reads "Step 1. Build more data centres. Step 2. 𒄑𒃻𒆠 𒁔 𒃲 𒀭𒉡 𒄭𒀀𒂍𒈠𒄭. Step 3. Profit", but would you spot the problem if the second step was "Use data centres to train AI model suitable for use in humanoid robots; I'll need some humanoid robots to test on, you should expect 50% of these to get broken in the process"? (Hint: I made that percentage up).
On a large scale, we have lots of examples of businesses or governments with bad plans that people don't realise are bad until much too late. Sometimes this destroys the company or brings down the government. Napoleon and Hitler both expected to be able to defeat Russia, the Confederacy expected to have no difficulty remaining independent from the Federates, Louis XVI and Charles I weren't expecting to lose their heads. Other times it's a much smaller problem, and we invoke "groupthink" to explain why none of the experts spoke up in advance to warn that the Bay of Pigs invasion wasn't going to work.
On the plus side, a human-level AI is also likely to be simply wrong about an explicitly evil plan it devises. On the down side, it's also quite capable of being wrong about a supposedly friendly plan, and we're quite capable of missing that error.
Plenty of people are currently loudly saying this kind of thing about half of Musk's projects. Are they right, or are they as wrong as the people who said electric cars would never beat hydrogen or the people who said reusable rockets would never bring the price down as far as the Falcon already has? It doesn't matter which answer you pick, as the point is that it's hard to tell.
When it comes to AI, a few sci-fi memes seem to be widespread: Skynet, the ultrapowerful AI god that manipulates humans into destroying themselves; the AI assistant that is so charming it replaces real human interaction, à la Her; even the term AI itself largely seems like a sci-fi inheritance, not an accurate label for the tech involved.
None of it is very connected to real-world developments, where things like Siri or Alexa are mostly just treated as voice-activated command systems, not real people.
And yet the technologists creating this stuff seem very influenced by what are ultimately implausible fictional stories. It's an unfortunate situation and I really wish the philosophy of technology / other more insightful analyses of technology were more prevalent.
He seems like the sort of person that would say everything is impossible right up until it happens. I hate working with naysayers like this.
The other day I was thinking... "We should do every thing we can to ensure SoundTransit Light Rail (in the Seattle area) should fail. That way we could replace it with something that actually worked."
This kind of thinking may be accurate, but it introduces near-intolerable levels of uncertainty for the future. Or maybe I should just give up on using mass transit to get to work in the Seattle area. Similarly, maybe I should give up thinking there's anything that will prevent the rentier class from extracting value from the commons by way of broken technology.
Or maybe I'm just having a bad day.
At least you understand the economic reality; that broken technology is now more profitable than working technology.
Most of us cling to a 20th century understanding of supply-demand and value-add. We still think the giant companies and celebrity superstar "technologists" are trying to "make the world a better place", We have yet to really grasp enshitification - a fancier word for betrayal and burning the social capital of science and technology (and people's belief in it) accrued over many centuries.
You don't even need to be "political" about this, I'm just a computer guy who wants to understand why everything around me is fucked, despite so many very smart and hard-working colleagues toiling at this for 50 years.
We keep trying to fix engineering, but we're tackling the wrong problem. At some point the social conditions in which engineering takes place must be addressed. It occurs to me that our ideas about engineering may not scale beyond the initial industrial revolution when Telford and Brunel built bridges out of iron that people could obviously see and benefit from, or they collapsed into the sea. Nowadays its confusing to know whether a new gadget is a labour saving godsend or a weapon pointed at you to destroy your life and liberty.