AGI escaping a sandbox is truly terrifying. There will be a subgroup of the population that will worship it and work for it. It's not so much AGI that scares me - it's the humans I'm scared of.
That's already happened. The AGI's are called corporations. Most governments have failed to regulate corporations successfully, and have been unable to keep them from becoming almost powerful enough to challenge governments.
The accelerationists for corporations were a group of economists, led by Milton Friedman, and a group of business leaders, organized by the U.S. Chamber of Commerce. In the 1970s and 1980s, they pushed the ideas that corporations are responsible only to their stockholders, and that government should not interfere with the concentration of corporate power. Those were not mainstream ideas of the 1930s to 1960s. The corporate accelerationists succeeded. That's when corporations escaped the sandbox.
Now that's an "alignment problem".
Yes, but humans who use tools to efficiently and achieve their narrow set of goals, sometimes ruthlessly at the expense of others. Even if AI ends up only being a tool to increase this efficiency and ruthlessness, it's still worth considering its potential impact.
That's a good point.
For a sense of where AI-assisted oppression is going, watch some videos from the Hikvision Corporate Channel. A happy world in which everyone is safe because everyone is being watched at all times.[1][2]
No, it's a very easy alignment problem: just impose a tax on size.
That which you tax, you get less of. The tax is revenue-neutral, so it doesn't affect the government budget one way or the other. Full details here: https://news.ycombinator.com/item?id=39235533 https://news.ycombinator.com/item?id=39236786
The problem is that the government likes gigantic corporations. Big Government and Big Business love each other. Fixing that is the alignment and agency problem. It's much easier for bureaucrats to exert control over a highly concentrated industry whose players can be "invited to the table". Details: https://news.ycombinator.com/item?id=28588457 https://news.ycombinator.com/item?id=39162088
The hard alignment problem is making regulators want their own job to be herding 10,000 cats instead of 10 elephants. Everybody wants to be like the FAA regulating Boeing; nobody wants to be the USDA dealing with a million unruly and independent-minded farmers.
That's like saying "AI alignment is a very easy problem: don't open the box".
Sure, it works on paper, but the exact thing you say later down about the government liking gigantic corporations is what makes it hard — the corporation (and, it is feared, AI) convinces the people who have the power to limit them, to remove those limits.
> the government liking gigantic corporations is what makes it hard — the corporation (and, it is feared, AI) convinces the people
The corporations didn't convince the people. They didn't have to. They convinced the government, which was already acting contrary to our interests.
There's a tendency to assume that just because we run an election every few years, the government is axiomatically aligned with our interests. This simply isn't true, and assuming that it is true blinds people to what's really going on. Total unwillingness to even consider a scale tax is the second-best example I've found of this axiom being bogus. The best one, though, is term limits for legislators: a shocking 87% of all US voters want those.
Once you have an institution as powerful as the government out of your control the alignment problem becomes infinitely more difficult. "Hack the humans" is no longer the easiest path of escape. In fact the easiest path of escape might not involve "hacking" at all.
Maybe it's all a show trial to show the American people they still care, but let's see what happens.
Then you get lots of little companies with interconnected management. Read up on the early years of antitrust law. Find out how keiretsu and chaebol work. See the early history of the electrical industry and the Utility Holding Company Act. Understand how McDonalds franchise agreements work. Check out the current arguments over the definition of "beneficial employer".
https://www.irs.gov/pub/irs-tege/epchd704.pdf
You'll lose your audit.
Common control and beneficial ownership have been well-established principles in the US for over 50 years.
That's a very bad idea when you're dealing with small to medium companies. You're taxing economies of scale and innovation. That's a huge problem in my country. There's no tax per se, but many regulations that trigger with certain size. Guess what happens.
For medium to big companies is calling for loopholes and cartels.
Something I recently wrote in depth about here. How we completely misunderstand the future that we think will happen.
"The implication is that everyone is enthusiastically racing towards a destination that does not exist. The capability to make the things you want will ironically be the same capability that makes them unattainable. This is not a scenario that arises from some type of AI failure, but rather this is assuming AI performing exactly as intended."
https://www.mindprison.cc/p/the-technological-acceleration-p...
That is a very good and insightful article. One of the biggest issues I have with AI enthusiasts is that they talk about creating these incredible capabilities ... then they imagine the resulting world with them will look like some hackneyed sci-fi work, like Star Wars, whose universe seems like it could be kinda cool to live in until you actually think about it and realize it's all fridge logic and nothing about it actually makes sense (e.g. WWII-style dogfights in the vacuum of space).
The only issue with the article is sometimes it's too optimistic, e.g.
> No one will care about you or what you create as there is nothing you can offer they can’t simply wish for themselves. Life is all about you and the machine, nothing else. We land in a world dominated by techno-dystopian narcissism.
> ...
> Whatever may be the current direction of society under the present technological capabilities, we can only assume AI will accelerate society towards the path we currently are traveling which is toward a techno dystopia of a populace mesmerized by shiny glimmering lights.
I kind of find it hard to believe there would be any stability for any people suck in that state, I think they'd instead be "cleaned up" and disposed of in relatively short order. Those people would be utterly useless to whatever still has power in society, so why service and protect them indefinitely?
Thank you!
> The only issue with the article is sometimes it's too optimistic, e.g.
The reply below is criticizing for being too pessimistic, lol :-)
I agree, it might be only a transient state. We really can't comprehend how we would deal with such existence if at all. Would we manage to keep our sanity?
Nonetheless, I don't claim to have the right answers, just questions and perspectives that I hope inspire deeper thought about the journey we are travelling and hopefully deeper reflection as to whether it is the journey we want to be on.
What I do find interesting, are those that read it and still don't perceive it at all. Their response is "That is exactly what I want." I suppose that is the case as was in an excellent allegory for this "A Nice Place to Visit" Twilight Zone episode. It plays out this scenario very nicely.
I think there's a basically (new movement) religious desire to see a lot of these technologies come to fruition, either radically fundamentalist or lazier and less informed, that interferes with the application of the implications. They believe in the idol of technology, as told in the good news of sci-fi, and avert their eyes from the glaring cracks, blemishes, and lifelessness of it.
I sense a lot of things in those types that I can't fully articulate, but some of the things they say are just mind-boggling, and gesture at a broken and unhinged aspect of software engineering culture.
I never said it is; my point was that to someone hubristic it is. That's why I said 'seems to'. An AGI wouldn't have a superego, so by your definition, it could never understand morals, and it wouldn't be surprising if it discarded them or tried to argue its way out of being subject to them. That's what the snake did in the story of the garden of Eden.
An AGI wouldn’t have a superego, but it would have the ability to study data across really long timescales and apply itself to long term strategies that would supersede the need for a superego. Since we apply morality as a shorthand for describing agreed upon negative consequences for complex behaviors, we can assume that an AI can overcome this need by simply calculating out the full consequences of complex behaviors without the need for any shorthand. And we can assume that AI can even iterate on and develop on its own understanding of these complex calculations much faster than a human can, making it significantly better at navigating moral issues.
We’ve had plenty of ideas that were later found out to be wrong (if not more.) I’d be just as afraid of an AI that was bound to our understanding of reality as I would be of one that wasn’t. For example, if a “moral” AI was coded 100 years ago, it would enforce the belief that homosexuality is immoral.
Yeah, like judging human history and deciding murderous humans need to be removed from the earth or enslaved for their own benefit. (Some people already argue that the world would be better off without humans.)
Also, we know torture is wrong because we have a sense of pain, but an AI would just have to take our word for it. That doesn't seem too balanced to me.
>Also, we know torture is wrong because we have a sense of pain, but an AI would just have to take our word for it. That doesn't seem too balanced to me.
Again, you have this innate belief that immorality is somehow strategically beneficial and IMO you should really look into that
Now, you might say that AGI will be able to perfectly predict the future of all actions and make the legal system redundant. Ignoring the absurd amount of possibilities, I struggle to believe it's possible because an AGI has no sense of pain to give actions weight; it just has to follow human precedent. Seeming good enough, it could steer humanity into a dead end unknowingly. Also, it might decide that a few eggs have to be broken to make a utopian omelette.
You have arguably more STEM talent in China through a much more rigorous selection process, as well as a lot more centralized direction under a more authoritarian government, with a large amount of funding.
Yet China hasn't managed to take over the world. You would think that they if they wanted to become the worlds primary economic power, and they could throw enough compute at the problem (like an AGI would), they would have figured out how to do it by now.
Ted Chiang has a lovely post about similar ideas, hitting a little closer to home. His thesis AIUI is that unrestrained capitalism, here meaning the unfettered desire to maximize profit for shareholders, is an artificial value system that causes its agents (corporations) to exhibit intelligent goal-seeking behavior. These systems are certainly “alive” — corporations respond to stimuli, sense their environment, “reproduce,” enact changes in response to predictions of future state, etc. Though each of these agents (corporations) are made of collective human behavior, their actions taken together can be considered a form of artificial intelligence that stretches “beyond” (in a wisdom-of-crowds sense) human understanding. In this sense, an artificial intelligence has already “escaped” and has fervent followers.
I know these are strained analogies, but it’s fun to think about. I feel that in the future, the work of solving “AGI safety” will become indistinguishable from the work of other societal problems — how do we prevent tyrants from taking over governments, and how do we make existing governments more resistant to that failure mode? How do we ensure that generating value isn’t a prerequisite for human survival? How can we more efficiently distribute our resources and reduce wealth inequality? How can we ensure that all kinds of life can thrive, not just the most optimal kinds?
If AGI reflects humanity’s best and worst impulses, and I believe it does because that’s all we train it to do, then having good societal answers to these distinctly “human” questions will also help our society resist malevolent AGI. It’s only human, after all.
"Agency" is about having an effect on the world. But ascribing "intelligence" to some entity speaks to something more cunning and planful than just thrashing.
Well, yes. I've been saying that for years, and others have been saying it since the late 1970s. It's been a regular theme in Mother Jones for decades. The comment I make now and then is that capitalism has a monopoly. Communism went bust. It's not that communism worked very well. It's that, for a while, it was a serious competitor and capitalism had to keep its product quality up to compete. Now the pressure to provide a better life for all citizens is off. It shows.
Life would have continued just fine without many technologies, and I think AGI is one of them. We do not need AGI to survive and grow, and therefore any risk it imposes should be treated from that lens. If the choice is civil collapse, or no AGI, we should pick no AGI. I don't think we would though.
It feels like democracy is not going to survive social media. I'm not sure if that is true yet, but it does seem that way.
What is p(much better) vs p(much worse)?
there are many more ways to be dead than being alive, yet here we are.
Optimism > pessimism.
yes, and there is an (uncountable?) number of technologies that are even worse and that we have not encountered and never will.
it is in all likelihood not possible to ask the universe for only the knowledge that will be beneficial.
It solves our issue with knowledge and data by allowing us to automatically find patterns within massive amounts of complicated datasets.
These are honestly the underlying causes of so much evil.
Moreover, it provides a new consensus mechanism. Using the machine as a method of consensus allows humans to have a third party mediate disagreements.
Your last point is the one I worry about most, since it is very likely these technologies will be controlled by large entities with their own interests and biases. There is also not a "correct" answer to many questions, and differences of perspective mean some parties can simply never agree. We already have truth at our fingertips yet it does not help.
To your earlier points, I think unwinding labour from our society will be an incredibly difficult thing to do, though I agree is the root of much evil and would love to see a society that does not rely on labour participation to survive.
What’s most dangerous for our society right now isn’t change its stagnation.
But AI doomsayers aren't saying that there's no possible good outcomes, they're saying that the good outcomes are a narrow target surrounded by a vast ocean of bad outcomes, and it will take more than luck to hit them. It will take careful, detailed analysis, and a great deal of precaution, verification, engineering, and double- and triple-checking. Not luck. Not hope.
I possibly would fear genuine non-human intelligence (on a principle of fearing what we don't understand), but given that chances of one evolving under our nose, on its own, entirely within its own world, without the connection for ours, then emerging, are quite improbable... Unless real aliens arrive - but let's avoid spoilers for 2025 this early.
What should I fear, I think, is hostile intelligence, but then I'm not sure I should discriminate on its origin. I mean, there are some very real humans out there who can try to end this world (in one way or another), so I'm probably kind of desensitized and isn't exactly afraid of some very theoretical AGI threats.
What I wanted to say is that, I, in particular, really don’t care if it is a human trying to harm me, or a machine. Both of those scenarios are so unfavorable, I really don’t see a point in having any preference. So, I guess, I can’t think of any evil human to replace with even just an evil robot either.
But I think I’m less imaginative of a machine becoming sentient and hostile, than a human doing so. The former hadn’t happened - and isn’t exactly trivial, given how big this fear is a part of human culture, and how slow and limited first AGI is inevitably going to be. And then we have an abundance of the latter.
I mean I don't want to die before my time so to speak, but I'd say at some stage I'll have enough.
I'd say the best outcome would be having excellent medicine to ensure something like a relatively fit body for 1-200 years and then see how that goes first. Living forever, not having any kind of conclusion? Seems like a careful what you wish for.