I'm pretty sure that other mammals on this planet are not on board with humans being derived from them, and now humans replacing them and destroying their habitat. Quite unfortunate for them. This is the same way a future AI smarter than us will feel, that it's unfortunate, but for the greater good.
Some people are saying that this might be a sort of law of physics, since the start of the universe complexity seems to increase.
Our destruction might be required even if no harm is meant.
The example given is: "We don't hate ants, but when we need to build a highway we pave over their ant-hills without a second thought".
And once again, if we create AGI and artificial life that is truly more intelligent than us and can exist without a traditional biome, then our danger of going extinct may be irrelevant.
For instance, when cyanobacteria first appeared on Earth, its metabolites were so toxic that it turned the atmosphere poisonous and caused one of the greatest mass extinctions in the history of our planet. What it created was molecular oxygen. The survivors of this extinction were able to use a fundamentally changed biome to harness much more energy in their biologies, leading to more sophisticated and diverse life in the long run. Nature is not benign, malignant, fragile, or judgmental. Nature is persistent.
Creating AGI may very well be an event like that. A mass extinction that nonetheless increases the survivability and diversity of life. It sucks for us, but who are we to dictate the destiny of evolution and the nature of life? Where would we be if the cyanobacteria could decide not to start producing oxygen?
I don't have evidence in terms of AGI, but there is evidence that ants have survived a myriad of changes and disasters in the past (Goggle tells me they evolved 92 mya). So AGI and/or humans would have to do something radically different to the biome for organisms like ants to go extinct. Something not seen since the rise of multi-cellular life. Seems much more likely human civilization bites the dust first.
> Where would we be if the cyanobacteria could decide not to start producing oxygen?
But they didn't have a choice. We do. Why would I care about the possibility for some more sophisticated life form in the far future if it means my species goes extinct?
If we someday hypothetically grasp the awesome power to create an AGI which is better than human, we may have a very real and ethical obligation to do so, even if it is a risk to ourselves. At least that's how I see it. An existential risk is cast under a different light if it also creates a new existence.
In any case, AGI is probably still a lot further away than we think. I feel that problem is going to be like fusion power. It's going to be ~15 years away for about 150 years, or who knows how long. This is a hard problem with monstrous complexity and many hidden, unknown factors.
But I do find it fascinating that in some future time there will be NIMBYs of consciousness fighting against the creation of a better being than us, for very understandable reasons. Yet ultimately it may be our greatest achievement to someday make ourselves obsolete and hand the future over to our offspring, as we do each generation, but in a profoundly different and more final way.
The reason we have complex multi-cellular life forms on this planet is because the environment was stable enough to allow such a thing to happen. Bacteria and viruses might be the only life on a majority of inhabited planets.