Understanding the Danger from Strong AI

> Given some random goal or goals, if you let imaginary humans, even bad ones, with unlimited capabilities optimise the world accordingly, the result should be compatible with human existence, because as humans they will respect all the constraints that go without saying, like that the atmosphere should be breathable.

Maximizing profit does not have anything to do with a viable biosphere. Obvious example is plastic pollution and its effects on biological functions. Cancer incidence is higher in areas with large amounts of synthetic chemicals in the soil and water. The companies producing synthetic chemicals are large conglomerates of people but their overall chemical metabolic output is detrimental to people.

Essentially, any argument that assumes people optimize for a viable biosphere is generically invalid and must contend with the fact that maximizing profits has ended up with the current ecological disaster. There is no need to invoke misaligned AI for ecological destruction, profit aligned corporations are more than sufficient. There is no point in worrying about super advanced intelligence that suddenly makes people extinct because the slow deterioration of the ecology necessary for human life to persist is already underway and AI/AGI/profit-maximizer is only going to accelerate that trend.