Hacker News Reader

Comment by stupidcar | Hacker News Reader

If you want an inadvertent AI doomsday scenario, how about black-box trading models that figure they can make money by betting on a market crash caused by a war, then manipulate the market to induce an economic conflict between different states, without every really having any understanding of the meaning of the outcomes they're optimising for.

View on HN

Better yet, how about black box recommendation models that figure that people are more likely to click on things that outrage them, who don't realize that the end result of viral fake outrage can be war?

But that one would be current reality. People's tendency to click on things that outrage them (and therefore generate advertising) is one of the things that lead to social networks becoming echo chambers of various conspiracy theories. The result is sites like InfoWars, which helped make Donald Trump president. The war bit hasn't happened, but it well could.

My favorite book about this topic is _Trust Me, I'm Lying_. It lays out the human incentives that guarantee that the recommendation engines will produce terrible for us results.

> The war bit hasn't happened, but it well could.

Give it time.

AI doesn't work that way.

Not yet.

You might find the history of Doug Lennat's Eurisko informative.

"Lenat fed the massive Traveller rulebook into the system and asked it to find the best way of winning. Each night, after several hours of trial and error, Eurisko would spit out a few strategies—some interesting, some impractical, some ridiculous. At one point, Lenat remembers, it suggested he could win the game by changing the rules."

https://www.wired.com/2016/03/doug-lenat-artificial-intellig...

ah this was Traveller book 5 Trillion Credit Squadron Publish in 1980 or so.

The authors commented that having a computer (when Pc's where ultra rare and expensive) would be useful :-)

I commented at that time that a PC was essential and working at the cutting edge of AI would be "useful"

Traveller (book3) was the basis for Elite as well

Bear in mind that Lenat won't release the Eurisko source code, and some people have speculated that it didn't do all that he said it did.

I used to work for Doug and let’s say there is some “dramatization” in that characterization.

Also Eurisko and other symbolic/semantic systems are fundamentally different from the NNs popular today.

AI will eventually be better at doing what the best people do.

Although I agree with this sentiment normally, this doomsday scenario would require a great degree of cunning and deception. An AI can succeed without practicing this kind of manipulation. I'm certain other humans/AIs would be able to identify this and build around it (or maybe even pull the plug)

The scenario may work with a human actor/corporation working with the AI, but then it's no longer that cyberpunk (and could probably be done with a large enough grid today).

AI is generally strictly bounded. It only looks at certain data and can only take certain actions based on that data.

Which is how I feel about most of the supposed Ai paradoxes like "The stamp collector problem" or even "Grey goo". They're like a 10 ten ton block of ice in the middle of a sun. Sure, it's physically plausible, there's nothing in the laws of physics that says it can't happen. It's only that there's no path leading up to it happening.

I think grey goo is fairly realistic. Having small, self-replicating machines has many useful applications. Flaws during reproduction could trigger a runaway replication scenario.

The latter already happens with the existing small replicating machines that we know. We usually call that cancer.

Life is already tiny self replicating machines, it still can't replicate inside of a wood wall very well. Further, life and nano-machines are stuck with the same building blocks. Outside of organic chemistry there are not a lot of options for self replication. At which point your nano machines are also just food for existing nano-machines with billions of years of evolution behind them.

TLDR; The problem is chemistry not design.

Wood wasn't digested by anything for millions of years. It seems not completely impossible that whatever we use to create the nanomachines wouldn't be eaten by anything. But of course you're right in stating that it's unlikely that we'll produce machines that are capable of self-replicating using virtually any available substrate. So the runaway scenario where the whole Earth is turned into grey goo is unrealistic.

Small bits of Wood get eaten by any number of organisms and much faster when wet. https://en.wikipedia.org/wiki/Dry_rot So, it's probably a poor choice for building blocks if you don't want the robots eaten especially if you want the to move around..

I was referring to the carboniferous period where literally nothing could digest wood, wet or not.

> where literally nothing could digest wood

Yea, that's not actually well supported. There is no where near enough coal for all wood over that time span.

Larger creatures like termites where probably missing. But, micro organisms don't eat dry wood on their own very quickly.

AI doesn't actually know what it's doing. It just optimizes for hidden patterns.

I wrote a whole blog post on this but tl;dr AI/ML can do unethical things without anybody knowing.

Of course, the trading bot AI assumedly would only have outputs as either buy or sell shares. But what if the AI after a long time of analysing news and correlating it with stock prices determines that the best returns are somehow by buying and selling in a way that crashes the market temporarily.

I guess the central point is that even if they aren't advanced enough to enact OP's scenario, they do not understand negative repercussions outside of their specific domain and so can unscrupulously cause havoc by proxy once they get smart enough.

AI works exactly that way. Reinforcement learning involves training a model using some input features, some outputs, and a value function (usually, how closely the output(s) match a known value in the training set).

Consider a model whose inputs are market prices, outputs are market transactions, and which has a naive value function that simply optimises for the amount of profit is trades result in in a relative short space of time. It's plausible that it could detect trading patterns that correspond to market crashes that can be exploited for profit.

These trading patterns might actually be causing or supporting political / economic shocks, such as bubbles, corrupt or unethical behaviour or inter-state conflict that then cause these crashes, but, to the model, this would be invisible or irrelevant. It would learn to perform trades that encourage these same shocks (e.g. undermining a large employer, causing job losses that cause political anger), and profit from the consequences. All without ever "understanding" what is happening outside of its model or having malevolent intent.

Unfortunately, humans do.

Winning on other people’s disgrace? Already happened, The Big Short, etc

Accelerando?

Reply on news.ycombinator.com