Hacker News Reader

Comment by acjohnson55 | Hacker News Reader

Something I've been worried about for years is the proliferation of technologies that give ever smaller numbers of people with smaller amounts resources the ability to impact billions of people, in potentially irreversible ways. Combined with automation, this is a recipe for disaster. We're clearly lacking in the ability to regulate and govern ourselves, and real world market forces (as opposed to the hypothetical free market) encourage reckless aggressiveness at the level of companies and countries.

Although, I must admit, as a doomer, I'm pleasantly surprised things aren't already worse than they are.

View on HN

This is what AI safety activist Connor Leahy calls offense-defense asymmetry. Imagine if a nuke could be created with household materials. We got lucky that wasn't the case, but we didn't know that it wasn't the case before studying physics. If we weren't so lucky that physics happened to turn out that way, we already wouldn't exist, because the offense would be so much stronger than defense, we wouldn't stand a chance even with the best surveillance systems.

A profitable prediction markets trader who goes by the alias semiotic rivalry posted this prediction on his X account recently:

> Prediction: in 1-2 years, almost everyone will agree that open sourcing the best AIs is totally crazy as the AIs become capable of autonomous hacking of most humans and institutions, serious terrorism and creation of biological weapons, etc.

And I mostly agree with him. As much as it's awful to have such a concentration of capabilities in the hands of a few governments or corporations, as far as I can tell there are no easy answers here, only a matter of trade offs and different risks.

Keep in mind the world is run by hierarchs. Of course the solution will look hierarchical in nature. The problem is fundamentally speaking, the hierarchism is a self perpetuating problem, because we still have to solve who amongst us can be trusted with the top of the hierarchy, which hierarchism states by definition "most of us can't be". It's like encryption, it's excellent at turning every other hard problem into a key management problem.

Non-hierarchical structures (like anarchy) are plausible only when defence is stronger than offence, because the individual/clan/nation can defend against any offence from any other entity, while also being incapable of attacking any other entity due to weak offence relative to the other entity's defence.

But reality doesn't owe us this luxury, unfortunately.

Offense increases entropy. It's easy to destroy. Attack surfaces are numerous. Humans are fragile. It's hard to defend.

Im fairly certain that the knowledge necessary to build a nuclear bomb is far less well-protected than the general public is led to believe. The only part that is actually difficult is obtaining a sufficient quantity of enriched uranium without somebody dropping a JDAM on your lab.

If the Manhattan project could build a fat man in 4 years your state university's physics department has somebody who could reproduce that given the necessary resources. It might take them longer than four years given that they are not staffed with the modern equivalents of Oppenheimer, teller, et al but they could do it within a human lifetime if they actually wanted to cause a mass-casualty event and weren't afraid of the aforementioned JDAM.

> proliferation of technologies that give ever smaller numbers of people with smaller amounts resources the ability to impact billions of people

did you mean greater number of people with less resources?

EDIT: thanks to others that point out how you can interpret this differently, which I think is: "it is now easier for smaller teams with fewer resources" rather than my original interpretation of "power is now concentrated so that fewer people can do this than before"

I had the same reaction, but I think they meant fewer as in "requires less consensus-building to wield meaningful power." Used to have to convince lots of people to go to war in order to kill a couple million people. Fast approaching a one- or two-man weekend project.

No, I think they mean that it used to take a giant lab full of specialists with lots of funding to do lots of damage before.

Now, the worry is a lone nut in a basement with no money could pull off an act of bio terrorism.

Fewer as in the size of an individual group, even though the total number of people who could find themselves in such a group is greater.

I’d say that is almost the definition of technology: that which allows doing ever more with fewer and fewer people and resources.

And technology doesn’t really care what you use it for. It can be constructive or destructive.

It “only” enhanced local destructive potential, until last century. Then it got to the point where a huge effort could produce a truly global catastrophe. And then a bit smaller effort. And technology continues to march on.

I think you would get a kick out of reading this paper on this exact topic, called the Vulnerable World Hypothesis https://nickbostrom.com/papers/vulnerable.pdf

However I think the result is disappointing, it concludes the only solution is a massive preventative surveillance state and global governance.

Every person's utopia is someone else's dystopia and vice versa. What you've described is clearly some peoples (ie. already billionaires) version of the best possible world. Not so much for the rest of us though...

I'm so surprised that HN is this much anti-government. I'm a doomer myself, I fear the rich, I fear the terrorists, and I think the only one that

  - can save my life, 
  - is interested in saving my life
  - i have some control over

is the state.

Yes, they are currently snatching kids of (illegal) immigrant parents and it is horrible, but who else would protect me against lone nuts in a basement, or robor armies of trillionairs?

My comment may have been misinterpreted, it was not meant to be anti-government. It is meant to be anti-centralized control though, regardless of the form it takes.

Reply on news.ycombinator.com