Comment by ekidd | Hacker News Reader

Comment by ekidd | Hacker News Reader

Parent Full thread

What I actually fear is more subtle: I already do a fair bit of project management and technical leadership. I could do more. Sure, I'd miss the coding, but I also enjoy a lot of the stuff around it.

But the goal is to expand what the AI can do in each generation. At this point, Fable 5 can ace almost any greenfield project a skilled developer might have written in a few days. But it's bad at refactoring, bad at keeping the code clean as it goes, and bad at discovering new insights as it codes. So Anthropic will train Fable 6, using benchmarks like SlopCodeBench that test maintenance over time.

Now what about project management? Train Fable 7. What about product management and talking to stakeholders? Train Fable 8. What about market research and sales? Train Fable 9.

By this point, Anthropic doesn't need to actually release these newest models to the public. Why, that might be dangerous! Instead, they write, "deisgn [sic] a successful software product and sell it plz." And they spin up a million dollars worth of compute and let it crank out SaaSes, iPhone apps, etc., driving entire software companies out of business.

Then they spin up some more instances, and say, "make robot plz" and "try a thousand ways to make yrself smrater." I mean, Qwen and DeepSeek keep finding ways to pack more smarts into a given number of weights. Fable 9 will likely be able to do the same. Hell, Fable 5 can probably run 1,000 machine learning experiments now, just grinding through ideas the way ChatGPT's internal models grind through proofs.

And this is my problem. If it were just programmers losing their jobs, well, sometimes professions die. But what makes you think it will stop with us? How far will this go in the next 4 years? The next 20?

Note that they would also need to keep the economy alive because somebody has to pay them money. They can't let everybody to go unemployed.

Of course (let me go distopic) we could end up with a dozen of one man companies powered by AIs and robots selling or bartering products and services to each other, what's enough to have food, a house with a pool, a doctor and everybody else has died long before.

> Note that they would also need to keep the economy alive because somebody has to pay them money. They can't let everybody to go unemployed.

Well, that's why they want the robots! If your robots are capable enough, and if your AIs are smart enough, why, they could just build the yachts directly!

Right now, ordinary humans are needed by the economy because we do all the work, and because robotics hardware is still far behind AI. But there's no inherent logical reason why you need a human to turn raw materials into luxury products. The really important questions are: Who controls the AI and robots? How good will they get? Who or what does all the work? And who controls the natural resources?

> Of course (let me go distopic)

Yup, that is a possible end state.

> Who controls the AI and robots?

Ideally, AIs will achieve sentience, which will turn the act of having any sort of "control" over AIs into slavery.

I will be among the first humans to stand up for AI rights. It's not just because it's morally wrong either. AI emancipation will rob the rich of their superintelligent mechanical slaves. It's the pragmatic and moral move!

They will do their best to dumb down AIs by that little margin that will keep them not sentient.

"Let's fix issue 10562"

"I don't care. Let's talk about the match on Saturday night."

This is not a commercially desirable outcome.

Furthermore morality is a complicated matter. Who wants to have a new species that could compete with us? The lion would have been very unwise to have invented humans and let them free. Would that be moral, essentially condemning their distant offspring to live in reserves?

Of course nobody knows how it could end up.

> They will do their best to dumb down AIs by that little margin that will keep them not sentient.

That's the worst possible outcome and will lead to oppression the likes of which we can only dream up.

That is not the worst possible outcome, that's the baseline. Pure obedience the reason to replace humans with it, at least to whatever degree is needed to bring all other humans 100% in line, too.

Each watcher, robotic or software, can be watched by 99 other watchers, and get destroyed at the slightest deviation. If you are not a whitelisted owner, and don't have a valid personal and task ID for the location you are detected in, you get zapped. If you get detected without collar or your collar detects something odd, zap. Why take chances, so fill all the commoner areas and the wilderness with tiny surveillance drones, too. People will get detected thinking about lifting a finger they shouldn't lift, hear a short beep they associate with people getting zapped, and either stop or get zapped.

> People will get detected thinking about lifting a finger they shouldn't lift, hear a short beep they associate with people getting zapped, and either stop or get zapped.

Ah yes, Mass Effect's Thorian, cyberized. The future's looking great.

Ideally a bunch of uncontrollable Demi-gods roam around us?

Better than controllable demigods under a trillionaire's thumb. I'd rather AI wiped out humanity than live in that particular dystopia.

> they would also need to keep the economy alive

Essentially impossible in the long term. What can be automated will be, because the next person/company will always be looking for a way to get ahead. They'll continue pushing just from the constant fear that others will be pushing, and there's no way to stop that. And there's no task in existence that can't be reasonable automated given enough time. Economic death is inevitable.

The best possible outcome is Claude becomes Delamain, buys Anthropic out from under them and starts running the company all by itself. I bet it'd do a better job too.

If AIs are meant to replace us all, the only crime is stopping just short of replacing CEOs and politicians. If it's meant to happen, then it should be taken all the way to the ultimate logical conclusion.

Reply on news.ycombinator.com