Also pretty common with authors iirc - a spouse with an income that provides stability.
That said, also easy for people to use this as an excuse.
1 karma · joined October 28, 2025
That said, also easy for people to use this as an excuse.
The AGI is smarter than you, a lot smarter. If it's goal is to get out of the box to accomplish some goal and some human stands in the way of that it will do what it can to get out, this would include not doing things that sound alarms until it can do what it wants in pursuit of its goal.
Humans are famously insecure - stuff as simple as breaches, manipulation, bribery, etc. but could be something more sophisticated that's hard to predict - maybe something a lot smarter would be able to manipulate people in a more sophisticated way because it understands more about vulnerable human psychology? It can be hard to predict specific ways something a lot more capable will act, but you can still predict it will win.
All this also presupposes we're taking the risk seriously (which largely today we are not).
Most of the time a new virus is not a pandemic, but sometimes it is.
Nothing in our (human) history has caused an extinction level event for us, but these events do happen and have happened on earth a handful of times.
The arguments about superintelligent AGI and alignment risk are not that complex - if we can make an AGI the other bits follow and an extinction level event from an unaligned superintelligent AGI looks like the most likely default outcome.
I’d love to read a persuasive argument about why that’s not the case, but frankly the dismissals of this have been really bad and don’t hold up to 30 seconds of scrutiny.
People are also very bad at predicting when something like this will come. Right before the first nuclear detonation those closest to the problem thought it was decades away, similar for flight.
What we’re seeing right now doesn’t look like failure to me, it looks like something you might predict to see right before AGI is developed. That isn’t good when alignment is unsolved.
I’m also not a moral relativist, I don’t think all values are equivalent, but you don’t even need to go there - before that point a lot of what humans want is not controversial and the “obvious” cases are not so obvious or easy to classify.
It’s not an anti-goal that’s intentionally set, it’s that complex goal setting is hard and you may end up with something dumb that maximizes the reward unintentionally.
The issue is all of the AGIs will be unaligned in different ways because we don’t know how to align any of them. Also, the first to be able to improve itself in pursuit of its goal could take off at some threshold and then the others would not be relevant.
There’s a lot of thoughtful writing that exists on this topic and it’s really worth digging into the state of the art about it, your replies are thoughtful so it sounds like something you’d think about. I did the same thing a few years ago (around 2015) and found the arguments persuasive.
This is a decent overview: https://www.samharris.org/podcasts/making-sense-episodes/116...
We have an interest in not destroying our own environment because it’ll make our own lives more difficult and can have bad outcomes, but it’s not likely an extinction level risk for humans and even less so for all other life. Solutions like “degrowth” aren’t real solutions and cause lots of other problems.
It’s “cool” for the more extreme environmental political faction to have a cynical anti-human view of life (despite being human) because some people misinterpret this as wisdom, but I don’t.
The unaligned AGI e-risk is a different level of threat and could really lead to killing everything in pursuit of some dumb goal.
An unaligned superintelligent AGI in pursuit of some goal that happens to satisfy its reward, but might be an otherwise a dumb or pointless goal (paperclips) will still play to win. You can’t predict exactly what move AlphaGO will make in the Go game (if you could you’d be able to beat it), but you can still predict it will win.
It’s amusing to me when people claim they will control the superintelligent thing, how often in nature is something more intelligent controlled by something magnitudes less intelligent?
The comments here are typical and show most people haven’t read the existing arguments in any depth or have thought about it rigorously at all.
All of this looks pretty bad for us, but at least Open AI and most others at the front of this do understand the arguments and don’t have the same dumb dismissals (LeCun excepted).
Unfortunately unless we’re lucky or alignment ends up being easier than it looks, the default outcome is failure and it’s hard to see how the failure isn’t total.
Most of the AI concern that's high status to believe has been the bias, misinformation, safety, stuff. Until very recently talk about e-risk was dismissed and mocked without really engaging with the underlying arguments. That may be changing now, but on net I still mostly see people mocked and dismissed for it.
The set of people alarmed by AGI e-risk are also pretty different than the set alarmed about a lot of these other issues that aren't really e-risks (though still might have bad outcomes). At least EY, Bostrom, Toby Ord are not also as worried about about all these other things to nearly the same extent - the extinction risk of unaligned AGI is different in severity.
Out of the options to reduce that risk I think it would really take something like this, which also seems extremely unlikely to actually happen given the coordination problem: https://time.com/6266923/ai-eliezer-yudkowsky-open-letter-no...
You talk about aligned agents - but there aren't any today and we don't know how to make them. It wouldn't be aligned agents vs. unaligned, it's only unaligned.
I don't think spreading out the tech reduces the risk. Spreading out nuclear weapons doesn't reduce the risk (and with nukes at least it's a lot easier to control the fissionable materials). Even with nukes you can still create them and decide not to use them, not so true with superintelligent AGI.
If anyone could have made nukes from their computer humanity may not have made it.
I'm glad OpenAI understands the severity of the problem though and is at least trying to solve it in time.
It's uncool to look like an alarmist nut, but sometimes there's no socially acceptable alarm and the risks are real: https://intelligence.org/2017/10/13/fire-alarm/
It's worth looking at the underlying arguments earnestly, you can with an initial skepticism but I was persuaded. Alignment is also been something MIRI and others have been worried about since as early as 2007 (maybe earlier?) so it's also a case of a called shot, not a recent reaction to hype/new LLM capability.
Others have also changed their mind when they looked, for example:
- https://twitter.com/repligate/status/1676507258954416128?s=2...
- Longer form: https://www.lesswrong.com/posts/kAmgdEjq2eYQkB5PP/douglas-ho...
For a longer podcast introduction to the ideas: https://www.samharris.org/podcasts/making-sense-episodes/116...
Kids pick up on this hypocrisy and it can backfire: https://en.wikipedia.org/wiki/Reactance_(psychology)
Autonomy is really important - being empowered to actually make the good decision.
It's a matter of respect imo, adults often have too little of it. The best teachers I had, had a lot of it.
Kids observe, if you tell a kid to be generous but you behave selfishly the kid will assume you're full of shit and ignore what you're saying. The opposite is also true. Talk is cheap, what you actually do is what matters.
If you know the state of everyone else’s you could game this to be in a better position.
Sony sees this as a competitive threat and EU regulators are bad so don’t understand what’s happening - their restriction of the activision sale is itself anti-competitive because it locks in the existing business model (which is favorable for Sony) and reduces real business model competition which would benefit consumers.
All this focus on exclusives is dumb and part of the old business model.
Mods like this are awful - way too much power has gone to their head just because they happened to be the first to create some subreddit.
There’s also a lot of other commentary on his capabilities outside of that one account, so it seems likely to be true imo.
The thing the quest is missing is eye tracking, I'd guess because of the cost - but that eye tracking is going to be a really big deal for good UX I think (along with the much better latency).
Meta also has the issue that because they're targeting social (given what the company is) the value isn't really there unless all your friends have headsets too (outside of the gaming niche). This is really hard. Apple by focusing on the utility for an individual side steps this in a way where it's a lot easier to bootstrap imo, especially with the ability for them to rely on their existing platform.
The reason centralized ad-based companies are so successful is the centralization solves serious UX problems that are unsolvable by systems that are not centralized. You need a unified platform where users share an underlying system that stuff can be built on top of that they can actually own, where the details of the system are hidden and it's possible to actually get UX/discovery/auth that is just as good as centralized systems. It has to solve for the problems that lead to centralization. [0]
This stuff will only work when to an average user they have no idea they're even on a system that isn't centralized, just looks like a normal app. You can't get there without solving bigger issues upstream from the app itself. [1]
[0]: https://zalberico.com/essay/2022/09/28/tlon-urbit-computing-...
[1]: https://moronlab.blogspot.com/2010/01/urbit-functional-progr...
Probably best for reddit just entirely kill 3rd party API access and restrict the power of mods. Arguably they should have done that anyway like Twitter did rather than just set prices high to sort of get to that outcome anyway.
You can't change the behavior without fixing the incentives, you can't fix ad-model incentives by being a user within an ad company, the users should just accept that when operating in that space and try to operate in spaces where they can actual own the platform. If you build a company on some other company's owned API you're on borrowed time the moment keeping you around no longer benefits that company.
At the time mp3 players were a niche with crappy hardware and low capacity. The iPod was a lot more expensive and a lot better - it caused general adoption.
Similar for the iPhone - smartphones were a business niche and had crappy hardware, the first iPhone was triple the price and expanded smartphones to everyone on earth.
HoloLens, magic leap are niche products (the latter also being crappy hardware), meta’s hardware is better but still mediocre and constrained by their lack of computing platform. Apple’s hardware is a lot more expensive and if successful will expand the market to everyone in a few hardware revisions.
The hardware for ambient light AR also sucks, it’s why HoloLens, magic leap and such are not compelling. It’ll be a while before that’s a viable option (and why Apple is committed to pass through currently).
It’s a lot more than just a discord alternative since it’s an entire computing environment with built in networking and p2p IDs, but the chat is a good current use case.
The goal is to escape the kind of problem you’re describing by fixing the upstream problems that cause them.