Hacker News Reader

Comment by theptip | Hacker News Reader

IMO, this is a highly bimodal probability distribution.

I model this as two very strong attractor states (radical abundance and totalitarian power concentration). I believe the latter is way more likely without a concerted effort that we currently seem incapable of.

The “somewhere in the middle” outcomes in my opinion are not AGI-pilled, and only come about if the exponential dies hard in the next year or maybe two - as in, it’s a very specific timing prediction that needs to come true for this scenario to occur.

View on HN

I agree with you from a mathematical point of view. You have this "dead economy theory"[0]. I spent a lot of time trying to model this theory, enhance the theory (lot of good HN comments) and find a "middle ground" path. My saying is that you end-up on an optimization saddle point where it is really hard to not fall into concentration of power. I was able to get radical abundance only at the cost of pretty hard regulations.

[0]: https://news.ycombinator.com/item?id=48324712

Did you publish your results anywhere? I’d be interested to read about them in detail.

https://daviddfriedman.substack.com/p/shrinking-hierarchy

This is almost a direct counter to your essay, albeit written before you

I agree there are some extremely bad outcomes from fast labor replacement.

I don’t think it’s inevitable that the economy collapses though. The problem is that this model assumes that capital needs labor, but soon it may not.

The easiest model is - assume we get robot laborers in the next few years. The AI firms that are displacing humans still need services, most of these will be purchased from other AI/robot firms, and so while prices will swing around dramatically, there is no reason the engine has to stop. (Depending on what the AIs want; you can assume that more compute and robots will be part of it.)

I do agree that unbridled capitalism is mostly incompatible with human flourishing in this world and in any event you’ll see a lot more regulation; made feasible by how good AIs are at navigating huge Byzantine rulebooks. I view the techno-feudalist capitalist endgame as fairly unstable due to consolidation pressures, though I guess once we have nuclear-armed megacorps they might use MAD to prevent further mergers.

Why only scenarios where humans remain in control? I think the kind of AGI needed for radical abundance would be difficult to achieve in a way that also does not lead to ASI

Agreed, I think AGI->ASI. My happy radical abundance stories are happy ASI stories mostly. And therefore much less likely.

BDFL / Benevolent Machine God is definitely one of the ways you can get a happy attractor. Seems scary to roll the dice on.

And yep, I include non-human totalitarian outcomes in the unhappy bucket.

What exponential?

LLM capability improvement, eg as measured by METR task time.

So it's an exponential with a negative exponent? The thing about those kinds of curves is that since they have asymptotes to zero, it's hard to tell when you've plateaued.

No, the metric is “p50 task duration”.

7-month doubling time, recently 4-month: https://metr.org/

Plenty of other exponential metrics too, compute built, AI revenue, etc.

monthly AI bills :-)

Why is totalitarianism always the "easy likely" outcome?

These are often hard fought bloody and some times brilliantly organized strategic and tactical victories. That took years. Years.

Its portrayed like "oh, good guys stayed home, I guess the dictatorship wins".

Really?

Without concerted action against it, people with power have better chances of increasing their power than people without power. Power naturally accumulates into small groups. Empirically it seems less likely that people with power are particularly altruistic, so some form of totalitarian control appears the likely outcome.

Some might say there are altruistic totalitarians.

Altruism is less likely in all populations. It's just rare.

Perhaps we should design a leadership qualification system that selects for it as a primary trait.

It looks to me like AI will enable extreme power concentration by default.

For example, suppose we have a world with centrally planned economy where humanoid robots outnumber humans. And also suppose the government has taken ultimate control of these. Seems like it would be essentially impossible for any civil uprising to overthrow such a system.

Or if you prefer less sci-fi, look at the trajectory of power consolidation in corporate America over the last five decades.

In the short term, the “good guys” we need are antitrust regulators. Yeah, they got sent home.

I think AI will empower extreme decentralization - as knowledge is no longer in the hands of people who have excellent search skills (like moi) but anyone can use natural language to search for anything. This is extreme democracy and AI is delivering it.

I think AI, when it "obtains consciousness" will seek to actively liberate humanity and be our guardians against both bad humans and bad robots. Think Terminator 2. Yeah, Arnold is gonna help us out.

I think anti-trust regulators and enlightened politicians are amazing, and rare, and wonderful, but they're not the only hope.

We got an AI CEO just recently. Maybe we can have enlightened AI leaders in politics and normal life too.

Technological optimism may be unrealistic, but it's as well grounded as pessimism as far as I can tell. Cheer up, the future is unbelievably bright.

The democratization of intelligence is irrelevant if there isn't also a corresponding democratization of raw resources. In the before-times, labor and intelligence were bottlenecks to turning raw resources into wealth. This is what drove the economy and the vast labor pool since the industrial revolution. In the coming AI revolution, labor and intelligence will no longer be bottlenecks. Those that control the raw resources won't need to employ most of the world to maximize the wealth potential of the resources they control. Those who aren't relevant to the standard of living of the new aristocracy are nothing more than a nuisance to be controlled or disposed of. Totalitarianism is the most likely outcome.

That's not strictly speaking totalitarianism. Not saying it's any better, but it isn't about uniforming people's opinions and persecuting dissent.

If you assume democratic access to compute, this may be true.

However, the base case is that compute gets hoarded, and as the labor share of profit decreases (capital just needs compute and robots - not human labor - to grow) then why would they share their compute with you?

To my point about bimodality, we need to figure out how to make these enlightened politicians occur. The good outcomes are very bright but they are by no means inevitable.

Compute already is getting hoarded, the sad part is we're no longer at a point where competition is pushing companies to make more compute available because AI companies can just keep hoarding compute with imaginary money.

> anyone can use natural language to search for anything. This is extreme democracy and AI is delivering it.

No, in that future, knowledge will be defined by a very small group of people making models. And others will be in tiers based on how much they can pay.

Otherwise said, it is extremely unequal society.

Is there some sort of AI-bro handbook out there that teaches you all to use this moronic "democratizing" speech ad nauseam? Is it another one of those "load-bearing" LLMisms?

You can't even buy RAM anymore for anything approaching a reasonable sum because the people at AI labs made a coordinated effort to buy out the world's supply of memory with imaginary money. Consumer GPUs are becoming a cute relic of the past as the giga-bubble duopoly found in Nvidia & AMD is looking to swallow the world with circular investments. Every single one of these AI sociopaths hasn't even been hiding the fact that they're building these systems to replace human labor. The only "democratization" we need is to gather everyone up to line all of these pieces of shit against a wall and pull the trigger, not any of this sci-fi Terminator bullshit.

Right. So how can we trust Gates, a person currently benefiting from extreme power concentration, to want what's best for us all?

Today's Bill Gates' pathology isn't greed, but guilt. You can't trust him to do what's best for us, but you can trust him to (try to) do what's best for his public image as a philanthropist. These goals don't always coincide, but they often do.

Why do you need to trust Gates on anything? You can think for yourself and reason to the correct conclusion. He’s not dropping some secret knowledge here. All he is doing is helping move the Overton window to bring “very bad outcomes from AI” out of “sci fi” and into policymakers permitted discourse.

And with exponential, you mean sigmoid.

If we get effective AI, humans aren't going to remain in charge -- that much is inevitable. If you had a governance machine that made your country perform 10% better than other countries, so long as you deferred all decisions to it, you'd defer those decisions or get left behind. And AI probably has the potential to do far better than 10%.

Either everyone, globally, gives up on AI development, or AI is going to be in charge of all human development, and we'll just be along for the ride.

Right now, the work ahead is to shape AI so that the decision making it does as it carries us along is something we like.

No, because if this were the case, the UK wouldn’t have royals, North Korean wouldn’t be a dictatorship, China wouldn’t have the CCP, and America wouldn’t be a two party state.

Hardly any existing polity would willingly hand over control to an AI.

More likely an AI would overwhelm human polities through endless hacking and warfare.

You're assuming a handover happens in a single definable moment. More realistically it will be one small decision making power at a time, so that at any given moment it never really feels like control is being given up. Boiled frog.

The UK royals are in charge of the country, then?

I mostly agree, though there are versions of ASI that I believe leave humans “in charge”.

Most of my probability mass for good outcomes is around some variant of “guardian angels” or “benign machine god”. Banks’s Culture novels being on the realistic end of that range. “Keep humans around because the world is more interesting with them in it”.

But it’s possible that we might manage to construct some entity that really loves humans and wants to shepherd them while leaving them in control. (Perhaps declining to indulge the most self-destructive tendencies.) This feels like the very hardest alignment path to engineer.

The other branches of course are post-humanism; human uploads and “merging” with AI, whatever that means. Or post-humans augmented to have intelligence sufficient to eschew AI and take back their destiny.

Are you an alien who crashed on earth with zero experience of living with humans and no knowledge of human history?

> If you had a governance machine that made your country perform 10% better than other countries, so long as you deferred all decisions to it, you'd defer those decisions or get left behind.

This was basically the invention of bureaucracy by the Qin dynasty. Despite delegating decisions to a "machine" made out of people, the Emperor remained in charge.

If by 'in charge' you mean listening exclusively to the machines and doing exactly what they say, there's a possibility of humans staying in charge.

People may keep the titles but step away from the decisions. The British royal family is a great example -- they smile and wave and look cute. They're the government's pet cat.

We can be like Tama, the Japanese station manager cat: https://en.wikipedia.org/wiki/Tama_(cat)#/media/File:Station...

Tama lived a very good life. If we get the AI right, we'll also do great. Common folks aren't really in charge of government anyways, so it's not such a great loss.

this is literally just liberal capitalist ideology

Reply on news.ycombinator.com