First of all, for any non-trivial real-world problem (like e.g. doing a simplest of tasks while respecting human moral values), we do not, generally, agree on the variable (function, actually) being optimized. Nor can we even define it with sufficient amount of precision. This is the core of what's known as "Friendly AI" problem (also known as "AI control problem").

Secondly, optimization algorithms "have no hidden agenta and are predictable" only insofar as you're able to comprehend their implementation. Which, for anything more trivial than blinking an LED, you're most likely not. This is the core of what's known as "programs have bugs" problem in software engineering, and also of the "bugs are exploitable" and "programs can have hidden backdoors" problems in software security engineering.

This is probably the most inaccurate statement I've ever read.

Here are just some of the assumptions it is making that are flawed beyond repair:

- Algorithms don't have unexpected behaviour.

- Computer programs operating in real time are predictable

- We can agree on what to optimise for

- Checking the source code is always possible

- Checking the source code allows you to understand a program

The people setting the parameters may have hidden agendas. Or even not so hidden ones, like "make me rich at anybody's cost".

Those givens are far from trivial, and getting them wrong is dangerous. You may as well say "Given we can perfectly predict the market, it's possible to make a lot of money"

> Given we ... can check the source code

That's not a given.

The algorithm, the problem it is applied to, the variables to consider and to optimize, the variables excluded from consideration, and the dats sources used are all choices made by someone with some agenda. And they don't have to agree with you on their choices.

A cancerous tumor or virus is essentially an organically programmed optimization algorithm. I can think of a few things not to like about them.