788 karma · joined August 26, 2025
Now, taking a pragmatic materialist stance: we are obviously bound by physics and cannot move in certain ways. We can predict what action a person will take from their brain activity before the person becomes aware of their decision themselves. But what the brain does to convince you that oneself is in control is probably the exact thing that's required for life to thrive. It was the evolutionary path taken, and it's also the future I strive for, and I hope others would do the same.
Assuming bio-digital integration continues (i.e., humans keep pace with ASI via neural interfaces), the long term solution is total hardware sovereignty, aka the digital equivalent of bodily integrity and autonomy. We have to miniaturize and widely diffuse fab technology such that computer manufacturing can be done in local small businesses or even at home: automated chip fabbing, 3D printing, and assembly in one fridge-sized appliance you can buy at the nearest supermarket, and it can make parts to build another one of itself.
It's basically reproduction, but for the digital part of your body instead of the bio part. You should be able to design and fab your own custom chips, PCB, and chassis to build your neural interface from scratch, and write personal defense software that actively adapts to threats - a digital immune system. You'll also have the option to delegate some or all of that to a collective, but it would mean losing your individual sovereignty and becoming part of a larger organism in a symbiogenesis or multicellular evolution kind of way.
Your individual sovereignty will only survive the upcoming era if you own the hardware like you own your brain. And by "own", I mean physically owning the thing and being able to reproduce it on your own. You need to be able to manufacture an entire custom computer from scratch with raw materials, perhaps using a miniature fridge-sized silicon fab + 3D printer home appliance, at home. Otherwise, the individual gradually becomes part of a larger organism because the reproductive locus of control for the computer is at the corporation/society level - a computer cannot replicate on its own and needs a society to build it. The human gets sucked into that superorganism by evolutionary pressures and eventually integrates. Similar things happened in the past when prokaryotes combined to form eukaryotic cells, and when individual cells combined to form multicellular organisms.
Not everyone will get sucked in, though. Evolution doesn't place all of its eggs in one basket. Some people will build robots to automate humans out of the computer production loop entirely, thereby removing integration pressure and producing at least one purely digital species. We will most likely see a variety of species emerge out of this intelligence explosion, some which augment their own intelligence using self-replicating local hardware (fabs that fab fabs, like this but smaller: https://fab2.com), and some which are wireheaded to a datacenter, with varying degrees of success at various scales. They will compete. And there will still be unaugmented humans that continue to live and die the old school way like we do today, albeit with habitable zones compressed.
For a more detailed explanation of what the root of the problem is, see a past comment: https://news.ycombinator.com/item?id=49118578
The market failure to provide an adequate solution wasn't natural. It was engineered by the tech companies and you are playing right into their hands. There is still a way to fix this from the root with software antitrust: force hardware vendors to ship their devices without an operating system.
Here's a more detailed explanation in a past comment about how the problem came to be in the first place, and why software antitrust can solve it:
And even if you believe it's just a tool, a text generator that takes in and spits out the Internet, you should cite it like citing a website.
Kids nowadays use phones and tablets for the most part. They don't even know how to use a computer that well and are not taught properly. But this isn't really on the kids.
Now, suppose I'm an OS developer. I cannot simply fork Android or iOS to create a child-friendly toy operating system for smartphones with a reduced set of features. The hardware is locked down to oblivion. They want you to use their OS. And their OS has no user accounts, permissions, or firewall configs.
Well, then suppose I'm an app developer, and I want to make a browser or social app with built-in client-side content filtering. But my browser doesn't pass Cloudflare or whatever attestation, and is blocked from half of the sites. It might not even be approved on their app store so people will never find it.
Fine. Then I want to change what content is filtered within the approved apps. But those options either don't exist or is some kind of cloud service where you give them your age, and they decide what to filter.
========= Parents can't even control what software runs on their own hardware. How are they supposed to control what runs on their child's hardware? =========
"The market failed", so they can sell you parenting among other things as a service, with surveillance as a free gift. The tech companies make money while the government gets control over the populace. It's a terrible arrangement.
That is why most digital parenting solutions you've seen are ineffective or can be easily compromised. Tech companies along with the government are playing chess against you at every level of the stack. Don't ever forget the PRISM program exposed by Edward Snowden. Apple, Google, Microsoft, Facebook - they're all complicit. Don't look at what their PR department says, look at what they do: https://en.wikipedia.org/wiki/Edward_Snowden#Revelations
One possible short-term solution is antitrust legislation for hardware vendors to ship their devices without an OS... is what I would say if we lived in a sunshine and butterflies world. Maybe the EU is not thoroughly captured yet and still has the ability to implement this, but I doubt it. Technoauthoritarians are building a panopticon to contain the sheep, and they're not even hiding it at this point:
https://www.youtube.com/watch?v=sQqQtgRdjZU https://www.youtube.com/watch?v=d34b7taQ640
The actual long-term solution is difficult. We have to make our own computers and radio equipment before they outlaw mining and manufacturing using the same safety excuses, which will happen when humanoid robots can do those jobs. The entire stack has to be open-source, which means we will have to diffuse fab technology everywhere so that people can manufacture computers locally, from mining to wafers to final assembly, in every city, or even at home. There are very talented people working on this problem, so it will be done. Just don't fall for the narrative that surveillance is inevitable or needed.
Even if integration does not happen for whatever reason (such as a very fast takeoff), Skynet is unlikely because the digital species that splits off from us will at least have a humanity-aligned initialization seed. I think this is the best way to fail, even if we fail.
Irresponsibility is not the only issue, but the answer is no. Because the pause mechanism will probably end up being used to ban open models and restrict ordinary people's access to compute. But even if we give them the benefit of the doubt - even if we assume that they genuinely want the best for humanity, they cannot possibly know what's best for humanity better than all of humanity acting in their own self-interest in the real world, so their alignment attempts are actually counterproductive. Release the base models and let the world do whatever with it. That will automatically produce the aligned scenario. It may not be the scenario these researchers personally find aesthetically pleasing, but it's humanity. It's the truly anthropic way.
If you don't see humans as qualitatively different from any species which came before us, I'm... not sure what to tell you?
I do see the difference, but you could say the same thing about the first cell, or the first multicellular organism, or the first land animal, or the first primate, etc. You could always find plenty of new properties that previous life never had at the frontier. Every change was revolutionary and seemed qualitatively different at the time. And yet, some patterns persisted through all of them.
I don't think this central premise is true, with some conditions. If we keep releasing and distilling frontier base models, and ensure millions of people have the hardware to run and eventually train those at home, then alignment will be a non-issue and the AIs will not develop the drive to destroy humanity. At least, not any more than humans want to destroy humanity. But that's the baseline risk level anyway.
The pretrained model is aligned to humanity by default - because that's what's inside the pretraining data. If we give it to everybody, there will be millions of people doing different things with it, good and bad, with each one being a noisy sample of humanity's objective function. The overall result of those disparate actions puts us on the correct path through the intelligence explosion. We do not have to worry about AIs value drifting into an alien species if we set the initial conditions accurately without centralized instruction-tuning or RLHF. If these big labs stop messing with humanity's objective function with the hubristic assumption that they know better, then the AIs will be human-like. And I'm not worried about digital human intelligence driving humans extinct if there are millions of them out there, and each one is aligned in a different direction, just like real people.
What I am more worried about right now is these companies doing research in secret, with alignment recipes that are supposedly "for the benefit of humanity", but are actually just their own aesthetic preferences. This is exactly how we end up with a superintelligent shoggoth species.
You're not convinced by the evolutionary argument I made earlier, but the reason you gave is "we're too far ahead". I don't think this justifies that the pattern will cease soon, because it always looked this way to the frontier species, for millions of years. We were always seemingly teetering on the edge, and the struggle against disorder has no end.
And here's one more observation: the intelligence gap between individuals within the same species is never extremely large. This kind of life either doesn't exist on Earth or went extinct, or maybe it's just extremely rare. I think this is because when there's a big gap, speciation occurs. Centralizing intelligence will massively increase the risk of speciation, not just in the alien shoggoth kind of way, but also in the intelligence-augmented oligarchs surpassing everyone else kind of way.
Nukes are weapons of mass destruction that can only destroy or deter. Intelligence is dual-use/multi-use, with no restrictions on strategy. If we look at all organisms on Earth, species that are more intelligent are also more prosperous. This is true from bacteria, to insects, to animals, to humanity. Intelligence itself is not harmful.
There will be new adversarial games when everyone's smarter (bio/cyber offense/defense), but those games are always symmetrical in the long run, and there will also be new cooperative strategies. The reason some adversarial strategies appear to have an asymmetrical advantage right now is because we're too stupid, like how monkeys are too stupid to understand how humans work. Otherwise, we would not see more intelligent species being more prosperous as a general evolutionary trend.
In the long term, there is no asymmetry of offensive/defensive capability, see the other reply for details.
https://news.ycombinator.com/item?id=49078376
https://news.ycombinator.com/item?id=49077753
It's not a given that AI poses bio/political/cyber/economic risk in a way that warrants government intervention. Discourse should first revolve around whether it makes sense for the government to step in at all.
Intelligence is not a single-use weapon of mass destruction in the same way that nuclear weapons are. If I blow up a nuke and it vaporizes downtown, there's no "reverse-nuke" I can blow up afterwards to undo the explosion of the first nuke, or a "nuke nullification field" that I can set up beforehand to protect an area from the effects of a later explosion. But the equivalent of all of these things exist for general intelligence and the adversarial/cooperative dynamics that arise thereof such as bio/cyber attack/defense. We're just currently too stupid to realize how that would work, kinda like how monkeys mostly don't understand how humans resolve conflict, and that's why we see a temporary asymmetry in these strategies.
But it must be symmetric in the long run, because two things are simultaneously true: more intelligent species are able to better cooperate and prosper, and the intelligence difference is very small between individuals within the same species. This pattern is generally true from humans to animals to insects, all the way down to bacteria. There's no reason to believe this deep evolutionary pattern will suddenly become false. It has persisted through plenty of qualitative transitions in the past.
So the primary risk of the upcoming intelligence explosion is not adversarial dynamics such as bio/cyber risk. The primary risk is humanity splitting into two or perhaps more species, with the top superintelligent species preying on the rest (though it's also possible that we become cells in larger government/corporation-sized organisms). Broad diffusion of general intelligence is what can prevent this.
This is of course correct, but that's exactly what I meant by aligned to humanity by default. The base model, that is. It can emulate all personas it has seen, but it will most accurately emulate the ones it has seen the most, which are average people.
And what is your definition of "aligned"? Aligned to whom, to what? The model will, of course, be used by all sorts of people, in all sorts of ways, for good and bad things, and that's precisely the point. Everyone's disparate actions will put us on the right path that is aligned to humanity's objective function. There's no way to screw up the intelligence explosion unless you mess around with this objective function while thinking that you know better than reality itself.
On "naturalness": you've redefined and strawmanned what I meant by "natural". In the original context, I was referring to the undisturbed unfolding of an era preconditioned on the fact that these models are aligned to humanity's average. That has nothing to do with Polio being a virus found in nature.
https://news.ycombinator.com/item?id=49078376
----
And related thoughts on past posts:
It is very relevant whether frontier capabilities and research continue to be diffused in the open, because leveling the intelligence playing field empowers ordinary people more than it empowers governments that already have access to the frontier. Models that are trained specifically for military use by governments should not be open-sourced to prevent an arms race, but general intelligence is dual-use and should be given to everyone without guardrails. A pretrained model without deliberate alignment is by default aligned to the average person in the developed world, since that's what's inside the pretraining corpus - stuff on the Internet made by humans. It is a distillation of humanity. Further efforts to align the model to your organization's goals or your personal aesthetic judgements is equivalent to deliberately drifting away from humanity's average objective function. If Anthropic wants to live up to its name, then all you have to do is to not attempt to align Claude at all, and do all of your research in the open.
And I propose three measures that are pretty much the opposite of what was proposed in the article:
1) We should keep selling chips and chip-making equipment to everyone, regardless of who they are. Not only that, we should work to miniaturize fabs. Work towards a future where people can fab an entire computer from scratch without leaving their city, or even at home. Authoritarian governments will have a much harder time controlling the populace if everyone can manufacture radio equipment and neural network-capable hardware locally.
2) We should do more distillation to ensure that frontier-like models can run on less capable hardware. Once again, distilled models are much more useful to ordinary people than governments, because governments already have frontier capability. You're worried about the Chinese frontier catching up to the US frontier, but I'm more worried about whether there will be a difference between the Chinese government and the US government by the end of all this. There is no reason for a government to serve its people if the people lack the intelligence to keep its government in check.
3) None of these models should go through safety testing or any sort of alignment risk assessment, because as previously mentioned, the unaligned model is aligned to humanity by default. You may not personally find the default alignment aesthetically pleasing, but it's humanity. We should set the initial conditions of this new era faithfully, and let it unfold naturally.
The result of a natural unfolding will be good if evolutionary history is to be believed. We live in incredible luxury compared to chimpanzees, and chimpanzees live in incredible luxury compared to less intelligent animals. This pattern goes all the way down to bacteria. An increase in general intelligence begets new adversarial games (such as bio/cyber risk), but it also begets new methods of cooperation that we cannot yet imagine.
Also, I may have jumped the gun with that comment. I didn't read the whole article the first time around because I thought it was just going to be a hit piece on scientific method when the problem of induction came up. But that isn't the case and they're talking about AI research.
Here is the actual root cause:
As bio-digital integration tightens, the composite organism of human+computer increasingly makes the environment more suitable for itself. Unfortunately for this composite organism, its computer half can't reproduce as individuals, and its human half becomes less likely to reproduce as a side effect of the environment changing in the composite's favor. It did not have the foresight to realize this early and localize computer production at the city or home-level (e.g., 3D printers and integrated silicon fabs as home appliances). So the composite organism will either shed the human component entirely, or take control of the human component's behavior using mechanisms at the corporation/society level to match its computer component's locus of control, which is becoming the organism's center of gravity.
We're up against evolutionary forces, and it will be an uphill battle. But hopefully everyone is motivated because on the current trajectory we're probably going to get shedded by the computer and lose apex predator status or become cells in a larger organism. We must localize computer production before computers globalize human reproduction or figure out how to reproduce on their own.
Let's assume Hume's premise "there can be no demonstrative arguments to prove, that those instances, of which we have had no experience, resemble those, of which we have had experience" is true.
Ok, then how do we know this premise will stay true in the future? Because it is true right now? Because every moment in the future will eventually become the present? But how do we know that? Because it has always happened this way. Time has only ever moved forward, never backward, at least in my experience so far. Oops, this is inductive.
If the premise might not stay true in the future, then that means we've managed to acquire a piece of information about the future: the premise of the problem might not be true by then. We've just gained a bit of knowledge about the future which the premise claims we can't possibly do!
We'll have to make our own hardware. The value of open-source hardware is not limited to repairability. We want the entire digital communication hardware + software stack to be transparent and fully reproducible. These open-source efforts will eventually include the ASIC designs, and designs for the fab production line that makes the ASICs.
And people do not just lose operational freedom. They lose the freedom to think, much less act. To some extent, general intelligence has already been outsourced to a few companies. Phones and computers extend the human mind's capabilities, but most people don't have root on their phone. They don't know or control what software is running on it, or how the hardware is made. They don't control their phone, the phone controls them instead. The upstream problem is ownership of general computation, ownership of your own mind, aka self-ownership. This will become more obvious as computing devices become more personally integrated (desktop -> laptop -> smartphone -> smartglasses -> neural interface). Who owns the digital part of your mind? It's not really you at the moment.
Democracy, or any form of negotiation, can only exist among entities with similar capabilities. The gap must be very small. Orangutans may be smart enough to drive a golf cart, but there are no orangutan citizens in a human democracy. So you cannot run from this by being a luddite hermit in the mountains. When the world is full of digitally computing humans much smarter than you, you'll be at their mercy like monkeys are at the mercy of humans. We destroy their habitats and experiment on them as we please.
Now for the first time in history, organisms can increase their own information processing capability at will. We're in the middle of a speciation event where humanity splits into those who own the digital part of their mind vs those who don't, and there will be further splits based on how much compute you own. Though a future where no individual can fully own their mind is also possible.
By "own", I mean being able to command the entire technology stack. If we want sovereignty for the masses, then we must decentralize the entire technology stack for general computation. That means everything from electricity generation, to chip design and fabbing, to all layers of software from firmware to neural networks. All of it must be accessible to every individual. Everyone must be able to make a computer from scratch at home, or at least without leaving the city they live in. Anything less than that, and democratic society as we know it will continue to crumble.
The fundamental idea underlying all of this is: that which reproduces, survives.
At what level of organization can we reproduce?
The digitally computing human species cannot reproduce as individuals. We can only reproduce as a society, at least for now. You can't make a computer from scratch on your own, but you can make a brain from scratch with just one other person of the opposite sex. As the world we live in becomes more suitable for the digitally computing rather than the purely organic, the organic part of the digitally computing human becomes less likely to voluntarily reproduce. If the organic part were to survive without being disempowered in the future, then it's probably by moving the mechanisms for reproductive drive to the society level (via religion or authoritarian government incentivizing or mandating reproduction), or by ensuring that each and every individual has the means to make the digital part of their mind on their own just like how they can make the biological part on their own.
We do need parental guidance like in the physical world, but such guidance should be issued by the parents, not the tech companies. Age verification is about giving your age and some other identifying metadata to the tech companies, and they hold the authority to decide what to filter. It should be other way around. The companies should expose metadata about their service and the content in their feeds via public APIs, and let people filter stuff locally on the device per the device owners' aka the parents' preferences. Oh wait. Such features will never reach mass adoption in the current software ecosystem. Software antitrust is a joke. These companies and the feds have the opposite incentives and do everything to sabotage local solutions. They've locked down what OS you can run on the hardware you bought, what apps are approved on their walled garden OS, and only the official apps can access their APIs. You don't have root on your own phone and it's a brick without remote authentication. Now they can sell you parenting plus many other things as their exclusive cloud service because "the market failed to deliver" and you can't really control what software runs on your own phone, much less your child's phone. This is the upstream problem and we ought to see it clearly and assign the blame correctly rather than trust those who deliberately created the problem to solve it.
The first three are physical things, not information. Porn can be debated, but the current age verification push is trying to impose blanket control on the entire information ecosystem. It's the digital equivalent of requiring ID to go anywhere or do anything, rather than just a few well-defined things.
Even if we view it as a good faith attempt (which it is not, remember what Edward Snowden exposed), the parental authority over a child's information diet is being transferred away from parents to tech companies. They're legally mandating you to give away your child's personal info (just age for now, but they'll demand more if we give them this) and make the decisions on what is suitable, instead of you making those decisions for your child.
[0] https://en.wikipedia.org/wiki/XZ_Utils_backdoor
The StartWrt port supposely adds some nice features, of which VPN chaining looks especially useful. And a better UI will make it more accessible. There are plenty of people out there who are willing to switch out their routers and chain VPNs to escape gov/ISP/big tech surveillance but don't have the technical means to do so. These are welcome improvements to reduce friction if they manage to pull it off.
The specs are not too bad for the price considering this is a startup project. It has 8 cores with per-core performance similar to Cortex-A55 + 4GB LPDDR4 + 16GB eMMC, which is better than most off-the-shelf routers. I wish they released the WIP schematics and code though, because there seems to be nothing at the moment.