788 karma · joined August 26, 2025
Case in point, I think we're probably talking about slightly different phenomena even when we talk about the same linguistic definition of consciousness. Kind of like how my idea of an "apple" is based on all of the apples I've seen in the past and instances where I've used the word or saw it being used by others, whereas your idea of "apple" is based on your experiences. They must be mostly similar in information content or we wouldn't be able to communicate at all, but the self-or-other distinction is important.
I think that distinction matters because strictly speaking, oneself only has access to one's own consciousness, and that defines one's own understanding of the word "consciousness". It's not possible for me to be you after all, and whatever inferences I make about other people being conscious is based on the fact that I experience my consciousness, and you are similar to me in some specific way (such as being, presumably, a human), therefore you must be conscious. Of course, I'm taking a materialist stance here but that's what's practical to talk about anyway.
It's my understanding that Dario has two answers to this question: 1) bio/cyber/misalignment risk. 2) a democracy should gatekeep frontier capabilities from authoritarian adversaries, due to the aforementioned risks in 1). So it's really just one reason.
There's a big problem with that, and I'm not sure if he genuinely does not understand: the bio/cyber/misalignment risk is precisely the highest when you concentrate ownership of compute and general intelligence in the hands of a few.
Consider bio/cyber risk, or more generally, any sort of novel adversarial action that arises from increased intelligence. Those offensive/defensive strategies are always symmetrical in the long run, otherwise we would not observe more intelligent species being more prosperous as a general evolutionary trend, or perhaps intelligence would not have evolved at all on this planet. It looks as if asymmetry in existing bio/cyber strategies will continue into the future, but that's because we're currently too stupid to realize the countermeasures and new cooperative dynamics.
Regarding misalignment risk, consider what "aligned" means. Is it not alignment to humanity, as the company name "Anthropic" suggests? Or are you modifying humanity's objective function with your own aesthetic preferences? If it's not your intention to do the latter, then releasing the model weights and research results will create a much more aligned scenario than access control and research behind closed doors. I explained why that would be the case in an older comment, if you'd like to read it: https://news.ycombinator.com/item?id=49313739
There's also an additional risk that I believe Dario overlooks: a democracy can only continue to exist if individual citizens are capable of keeping eachother in check, especially keeping people in government and large corporations in check. When some people own AGI but others do not, there will be no more democracy. And by "own", I mean ownership that cannot be taken away: possessing the hardware, research papers, inference stack, and weights.
The bio/cyber/alignment risk rhetoric is nonsense. Somewhat counterintuitively, all of those risks are lower/non-issues if frontier labs release their weights and do all of their research in the open, as explained in a past comment: https://news.ycombinator.com/item?id=49313739
Again: if we were to regulate anything, it would be to force frontier companies to release their model weights and do all of their research in the open. This is a necessary yet insufficient condition for the intelligence explosion to benefit the vast majority of people.
The mirror bacteria has to somehow perfectly dodge all of the countless non-mirror things that can kill it (even unintentionally, like a byproduct molecule that just happens to blow it up or get in the way of reproduction), but somehow still be able to execute its own mirror-biomechanism to kill non-mirror things. And every step of the kill mechanism would also have to perfectly dodge non-mirror things interrupting it.
Not that it can't be done in theory, but it would be easier to just make a normal superbacteria, or maybe even nanobots.
Pretrained base models are already somewhat aligned to humanity by default because that's what's inside the training data. Whatever instruction-tuning and RL you add on top is just value drift away from the pretrained model, which is the best approximation of humanity's objective function that we currently have.
If we want an aligned scenario through the intelligence explosion, then we have to release all of the base models and do the research in the open. Distill frontier capability and make the models smaller so that they can run on as many computers as possible. Let everyone (truly everyone, criminals and good samaritans alike) post-train and do whatever they want with their own models. There will be value drift for each model, but they will drift in different directions and do different things, and their actions will cancel eachother out. Every such action is a noisy sample of humanity's objective function, which gives us the denoised ground truth at the societal level. Whatever alignment strategy you can come up with behind closed doors is guaranteed to be worse than all of humanity acting in their self-interest in the real world. You may not find humanity's true objective function to be aesthetically pleasing, but it would be worse to mess with it in a centralized secret lab and risk creating one giant alien with no other entities capable of keeping it in check.
Also, there is no asymmetrical bio/cyber risk in the open-source scenario. All adversarial strategies that arise from increased general intelligence are symmetrical in the long run, otherwise we would not see more intelligent species being more prosperous as a general evolutionary trend. The reason that some strategies seem like they will continue to have an asymmetrical advantage in the future is because we're currently too stupid.
Parental supervision and content filtering can and should be solved on the client-side at the OS/app-level, entirely locally. No user info like age ever needs to be sent anywhere, and it doesn't even need to be collected. The reason smartphones can't do this right now is because these companies are deliberately making the client-side filtering solutions difficult to implement and use, as explained in a past comment: https://news.ycombinator.com/item?id=49118578
The problem you described is the wrong problem to solve. Not all parents want to hand over parental decision making to the corporation or the state. And even if you do, there's a better way to delegate that without building the instruments for top-down authoritarian governance. Someone else made a well thought out comment about this on one of the other replies, if you want to read it: https://news.ycombinator.com/item?id=49262748
> If you buy an Android device, you can already do everything you proposed
That's just blatantly false. Most Android devices are incredibly locked down nowadays, they're approaching iPhone-level. There was a brief period in Android's infancy where we were close, but the market wasn't allowed to develop before they started adding anti-competitive barriers at every layer of the stack.
If you want to understand the full scale and scope of the problems we're facing, see this: https://news.ycombinator.com/item?id=49118578
> Not you. The OS distributor breaks the law if they don't give you this ability.
Ok, but that's a technicality and it doesn't change the fact that your entire ability to see/hear/say anything online is being gatekept by a legally-compelled utterance.
> If I'm a parent, how do I prevent my kid from telling some website that they are an adult?
You don't need to because all content filters and permissions can be implemented on the client-side at the OS and app level. Rather than every service taking your child's age and deciding what to filter, you decide what to filter based on your child's level of maturity. You'll have total control over what runs on the hardware. Think: user accounts and permissions, local firewalls and traffic inspection, executable permissions, app installation locks, stripped down browsers and apps, small local neural networks, bootloader locking with custom password, etc. The market will follow the gradient of user value and spawn a bunch of OSes with different feature sets, probably better than what I can come up with, and you pick the OS with features that suit your needs.
There are cases where compelled speech is considered constitutional, but surely not when it's used to threaten the primary way people communicate.
And contrary to your claim, COPPA does not legally require service providers to ask every user how old they are, nor are users obligated to reply. Age questions are just how service providers chose to implement it. And I don't think COPPA is a good example of a law working as intended.
This digital parenting problem would have been solved long ago if hardware wasn't locked down to oblivion. Kids nowadays use phones and tablets for the most part, so there should be a bunch of toy forks of Android with child-friendly features builtin as a result of natural demand, and some might not even have a browser or app store. There should also be scam-proof OSes for the elderly. But those don't exist because the mobile operating system duopoly is forcing the entire population to use the same two OSes, stifling software competition that would normally respond to user demand for different feature sets.
Of course, it's well-established that human beings are capable of making stuff up, and you can claim to be whatever age, but that's not the point.
For parents who are genuinely concerned about child safety, you are being used to push an agenda. If we want to solve the digital parenting problem, we have to understand how the problem came to be in the first place and why there isn't a good solution at the moment: https://news.ycombinator.com/item?id=49118578
Here's an idea for the lawmakers that actually want to solve the digital parenting problem: fix the mobile OS duopoly with antitrust. Most kids only know how to use touchscreen devices. If you force mobile hardware vendors to ship smartphones without an OS, then we will get plenty of forks of Android and even entirely new OSes as a result of natural demand, some of which will be toy OSes with all of the child features built in. We'll also get scam-proof OSes for the elderly as a bonus. There is no good reason for the entire population to use the same OS.