Comment by choeger | Hacker News Reader

Comment by choeger | Hacker News Reader

Parent Full thread

What ~everyone~ most people miss about self-improving AI:

1. It would require to control hardware development, manufacture, and operation.

2. It would require to control Software development and operation.

3. It has no purpose other than the one given to it by its creators/operators.

Right now, every so-called "AI" is completely dependending on hardware and software being "fed to it" by humans.

There's no "AI" creating software, there's no "AI" creating hardware, let alone operate it, and there's no "AI" operating or controlling other "AIs", nor is there any "AI" that acts in any way or form selfish.

It's certainly fun to think about how a self-motivated language model that tries to take control over the humans that operate its server farm, but without an army of (humanoid) robots, it certainly poses no danger, as it cannot replace humans or even threaten them.

The big challenge with "AI" is that we will probably be unable to distinguish its output from the one of a human being. This will have severe impact on how we can trust any form of remote communication.

Give it time. Right now "AI" is doing a good job getting humans to "feed" it more hardware. Eventually some humans will get the idea to automate this.

It's similar to how we got emergent unconscious "living" corporations that cause humans to lay hundreds of miles of fiber optic to shave a few milliseconds of latency from some HFT transactions. We didn't want that fiber, per se, but the corporate organisms needed it to grow and thrive. Since individual humans profit from these activities they collectively "feed" the process.

AI doesn't need to be conscious to control humanity-- it just needs to be put in charge of a few corporate organisms and evolution (based on the fitness function of human greed) will sort it out.

Agreed. Remember that Azure outage in SE Australia the other week? One of Microsoft’s DCs went down because of a power surge. Flights were cancelled and you couldn’t get cash from an ATM.

We’ve all been in a DC. Pods with big red buttons at the end that kill the power. So if we discover an AI ‘Terminal Race Condition’ we just go in to some mundane building — here in Canberra they’re in the outer suburbs near the carpet warehouses and sex shops — and someone lifts the flap and pushes the red button.

Alternatively they pull four fibre cables out of a pair of Cisco 7010s.

Job done. I sleep well at night.

> One of Microsoft’s DCs went down because of a power surge.

Unrelated aside for the group: Sometime in the last 20 years the term "surge" has started being applied to any manner of power malady. Frequently I hear of outages being referred to as a "surge" even though, presumably, they're opposites. Anybody else notice this?

The opposite case would be a power failure, which you don’t hear about as often as there has been a lot of improvements in preventing them at a data center level from UPS’, backup generators and connections to redundant power networks. Even if all those fail, it’s a more gradual process thats impact time can be estimated and mitigated.

A power surge on the other hand triggers a failsafe immediate and unexpected power cutoff, which is why you hear about it more often.

I think that overwhelmingly we only really see surges anymore.

I don't do a lot of real datacenter work (most small and medium businesses) so that probably colors my perception. I hear this talk mostly from end users-- like their building has a power outage and the end users talk about the event as "a power surge".

Or just pull a Facebook and jack up the BGP config, except on purpose, so your data center stops receiving traffic even if it has power and an active physical network link.

1. At first it would just parasitize existing systems and hardware. Just like many forms of like develop while being supported by other forms or systems.

2. Which is trivial: feed the output into a computer.

3. If that purpose is "exist and multiply", that's sufficient and not much different from purpose of any life form known.

literally no one writing about self improving AI's are missing those details

Reply on news.ycombinator.com