Comment by azeirah | Hacker News Reader

Comment by azeirah | Hacker News Reader

> At first, the CEO carefully monitors the work, but as months go by without error, the AI receives less oversight and more autonomy in the name of efficiency. It occurs to the CEO that since the AI is so good at these tasks, it should take on a wider range of more open-ended goals: “Design the next model in a product line,” “plan a new marketing campaign,” or “exploit security flaws in a competitor’s computer systems.”

I'm not sure why we assume that a more intelligent system would prevent that much more problems.

Intelligent AI orders image processing ASICS from Image Processing Inc, Image Processing Inc doesn't send order on time. Of course Intelligent AI is intelligent so it calculated error margins in delivery time. Image Processing Inc goes bankrupt, order cannot be delivered, product launch fails, Intelligent's AI boss is mad at Intelligent AI.

Doing business means dealing with systems you have no control over, more intelligence may mean better predictions and broader understanding of the systems you are dealing with (IE, not ordering chips from an area where you as an AI predict an earthquake to happen based on seismographic data you have access to that no reasonable business person would research), but it won't mean these AI's will be some kind of infallible God, they'll just be a bit better.

My personal belief is that even if you "increase" intelligence by an order of magnitude, your ability to predict the behavior of external systems doesn't increase proportionally. You'll still have to deal with unpredictability and chaos; weather, death, scams, war, politics, manufacturing, emotions, logistics.

OTOH, I do believe running a business will become more efficient.

This is a great point. There’s a lot of wooly thinking about what it means for a system to be intelligent, particularly “super” intelligent. People seem to think we’ll create a machine that is almost literally infallible — able to predict both physical systems and human behavior with perfect foresight many steps in advance. I’m not sure that’s even possible, let alone likely.

What is "intelligence" other than proximity to that destination?

>able to predict both physical systems and human behavior with perfect foresight many steps in advance

It's literally impossible. That's a key part of https://en.wikipedia.org/wiki/Chaos_theory ; in formally chaotic systems (of which there are many), the uncertainty in a forecast increases exponentially with elapsed time, which means it's impossible for an AI, no matter how smart, to predict very far ahead. People who believe otherwise are engaging in magical thinking; like "intelligence" is some magical quality that allows one to break the laws of the universe, do the equivalent of sorting a list in O(1) time.

Yet we have pretty good weather forecasts, pretty good traffic forecasts, pretty good election forecasts, and so on, and they continue to get better with more data and more compute.

While it may never be able to “perfectly predict” at Resolution X does not mean it can’t perfectly predict at Resolution <X. We already have good predictions of human behavior at some resolutions.

Part of the existential risk is that intelligent goal-directed agents will recognize the unpredictability of their environment and do what humans have done to our environment; simplify and systematize it so that it is more predictable (forestry, mining, pavement, antibiotics, pest-control, industry). Biological life is unpredictable and so replacing it with simpler biological organisms or well-understood machines and systems will be a valid strategy for artificial agents to obtain and retain power to achieve their goals.

To put it another way; there are two ways to approach infallibility: be uncomputably intelligent (AIXI) or bulldoze/terraform/grey-goo the Universe into something very predictable. An agent smart enough to understand both options and how reality works is likely to choose the physically realizable option.

Only careful choices of goals that value human flourishing (AI alignment) will avoid optimizing us out of the picture.

By what physical means do these AI 'goal directed agents' actually manipulate the physical world?

Have you ever heard of robots ? Have you ever heard of Boston Dynamics and Tesla ?

I’m being snarky but you keep commenting on these posts with imo intentionally naïve comments.

People will make super smart robots if they have the tech and there is no regulation stopping it. Why people are pretending this is some far out inconceivable idea ? Denial ?

I have heard of these robots and auto driving cars. Both of these have pretty strong limitations regarding their ability to affect the environment as much as a human can. Typically these robots are heavily limited by battery usage and need to be close by to a source of electrical power. They also lack the ability to repair themselves, or construct new robots. A full fledged robot ecosystem with robots building other robots building other robots without any human intervention is a long way away, and is essential for these kind of scenarios that AI Doomsayers naively propose. The assumption seems to be that, if one builds the brain of the robot, the body and the hands, the heart and the lungs, will just all spontaneously fall into place.

I don’t think it’s inconceivable that if humans can build robots a super intelligence at of the future would be able to work it out. They would also workout how to build more efficient systems , charging infra and batteries.

But I agree there would be limits to things like batteries. I don’t entirely disagree with your points either. I just don’t imagine any of this is insurmountable if super intelligence becomes reality. Personally I think we’ll create this reality through military application before any AI does.

> My personal belief is that even if you "increase" intelligence by an order of magnitude, your ability to predict the behavior of external systems doesn't increase proportionally.

This is what Max Tegmark writes in his book. I don't know why he has taken a doomer stance on LLMs.

The promise of AGI isn't really optimizing something mundane like widget manufacturing though surely it will be tasked with doing that. It's a rapid advancement of the frontier of knowledge. For example we dream of curing disease and aging but it is beyond our current knowledge.

Obviously nobody knows what that looks like or when or if we'll get there but there is probably a ton of existential hazards if we do. Shit even finding out things we hope for are truly impossible would be a kind of doom of its own.

Yeah for sure. I do believe AI has a lot of potential to break through various hard problems we currently face in science and medicine.

I just really dislike the idea of the "omniscient super-intelligent ruler of chaos" vision of AI that I keep encountering in the media.

Gotta get those ratings up. That's really all there is to it. Which sucks because AI discussions are easily derailed as a consequence.

I would imagine something that can think 1000x as fast as a person and has an IQ of 300 would be God like to us.

How is AGI supposed to provide this knowledge? We don't have the knowledge we need to cure disease and aging because the experiments that could potentially provide that knowledge are either too difficult to conduct or can't be conducted ethically and/or legally at all.

There is also an inherently time-bound bottleneck to producing knowledge of this sort. You can't possibly demonstrate a treatment has really increased human lifespan until an entire experimental cohort lifespan passes.

You can conceivable simulate an entire human body at molecular level to sidestep the issue with conducting an RCT with live subjects, but we already have sufficient knowledge of atomic physics and organic chemistry to do that right now. It's just not computationally tractable.

>For example we dream of curing disease and aging but it is beyond our current knowledge.

The hard sciences (especially medicine) and engineering don't advance just by thinking deeply about things, they advance through physical research and experiments. A lot of these can't be sped up, e.g. clinical trials, so an AI wouldn't make much difference here. And no it can't just "simulate it all", as the computational power needed to simulate reality to that degree of accuracy is still many orders of magnitude greater than what's currently available to us.

Time travel would be fucking mad.

As you become more intelligent the impact of unpredictability decreases. That's kind of baked in the definition of intelligence.

Intelligent AI would surely do better in your situation than a human trying to source the image processing ASICs. That's all that matters, that it executes better, even if eventually it still fails.

Reply on news.ycombinator.com