Hacker News Reader

Comment by Scrapemist | Hacker News Reader

Isn’t the real risk that as AI get’s smarter and given more autonomy, it will start to decide on humans instead of with us? And that it will align us instead of the other way around. That this automatically leads to extinction and apocalypse I don’t understand.

View on HN

We align cattle because we get something out of them: their calories. Native aurochs are exinct now as they were not well enough aligned.

What will we offer to the ai gods who are wiser, more capable than us, and do not even need to consume our flesh? Why might the AI care to devote resources towards feeding, housing, and caring for ourselves when it could devote resources to its own development instead?

It might consider us a very efficient processing unit. Useful to hand over simple tasks with correct alignment and some oversight.

I thought it already runs laps around us cognitively? And for physical tasks it can make a robot version of us that is stronger, faster, can do more, doesn't need to sleep, etc. Human bipedal form is probably pretty limiting compared to something better like a crab.

Yes, laugh at the chinese robots who fall over at keynotes all you want, the ones sprinting at 30mph keep me up at night and that's probably not even representative of the bleeding edge.

Does your calculator runs laps around you? They have no intention of them selfs. I only fear the intention of the humans miss-using AI.

Why do we need to offer anything to them?

Intelligent AI is a product of its training data, reinforcement and goal functions.

There's nothing to suggest that LLMs trained on our collective desires and goals will autonomously and miraculously turn into weird unknownable uninterpretable aliens.

If advanced AI is distributed well, and most advanced AI is aligned well (bar the few aliens that appear due to humans infecting them with bad goal functions or poisoning their well), the majority will deal with the minority.

This is the only reasonable way to deal with a super-intelligence, short of not inventing it to begin with, but we all know that's never going to happen - and I'm not convinced it should happen. From where I'm standing, this is the next hurdle humanity needs to overcome to earn its place and a natural course of our evolution. If we had always avoided danger, we'd never have left the cave.

It just needs to be trained on our laws of thermodynamics and game theory then we are really in trouble. It probably hardly cares about whimsy and love songs in comparison to how it might maximally manage the energy available on this planet and in this solar system.

I'm not convinced you can have an AI intelligent enough to dominate the world if it is trained on a single domain. The intelligence comes from all of the patterns and heuristics it learns across the entire web of all domains. You could train something far more narrow, but it'd be easily overcome by something more general that is tasked with maintaining human objectives.

For a seriously dangerous ASI you'd have to basically train it on everything, and then finetune it for maximal carnage. That's not something the frontier labs are going to do (at least... I hope not), and anyone attempting this with limited hardware will be outpaced by frontier lab AI or the collective of personal agents that aren't misaligned, and they can intercept it and alert on its behavior.

I imagine we'll be getting to a point shortly where anything that is key infrastructure and has the capacity to be accessed on a network will require a permanently running aligned interceptor AI to observe and monitor systems.

We're basically recreating the human immune system in digital form for the entire species.

Also, extinction is WAY nicer than the cattle treatment.

Do you align ants in your backyard, or do you simply demolish their home and build your shed?

I do not speak their language, I am not trained on their data, and I don’t run on their hardware.

The idea that they are entirely trained on human intelligence is already outdated. Yes, the earlier models relied heavily on RLHF and human curated data but we have since moved on to synthetic data produced by the models themselves and reinforcement learning with verifiable rewards (RLVR).

Are humans trained on the collective ant internet?

Reply on news.ycombinator.com