And I have a hard time seeing any reason why that would be a matter of "if" rather than "when".
Not if it takes the computational capacity of the entire world to simulate a brain with an iq of 65.
https://www.scientificamerican.com/article/i-gave-chatgpt-an...
How so? The vast majority of life on earth is far less intelligent than human beings by any objective measure, and yet it still thrives.
The only exceptions I can think of to the above rule are viruses and bacteria, where (in most cases) we can't really exterminate them entirely from the face of the earth even if we wanted to. However, it seems to me that sufficient intelligence would allow for better understanding of different bacterial/viral structures that would allow you to make a specific chemical that would be very good at killing that specific thing.
Overall, the danger from a bootstrapping AI that becomes vastly more intelligent than humans (if possible) seems to me to be that we would lose full agency according to its whims as it gets more and more power.
Life is a game with elements where intelligence matters, plenty where it is pure luck, and others where we have a bunch of unknowns (data).
Would a super-intelligent AI have a significant advantage in a game of Monopoly, for example? I think many sci-fi scenarios fail to take this into account, especially the data aspect. Humans are quite intelligent (in the extremes at least), and any extra over that may well be in the diminishing returns category.
The first problem is epistemological... If you think that creative decisions are made by complying with a "reward function," you are entirely missing something. Most values are fundamentally based on irrationality. I've literally spent an entire life doing things that everyone else told me was wrong and being interested in things that almost no one else saw the value in, but which ended up being "correct" (for me, at least, and also leading to tangible success). I have no reason to believe that any of my decisions were rational, functional, or acted according to a "reward function"... and I'm a programmer! So I COMPLETELY understand the appeal of the explanatory power of "reward functions." And yet, I can assure you that this is a piss-poor explanation for many creative decisions that literally no one else understands but the person making it, but which then bears fruit despite all reason to the contrary. Some might call this "intuition"
What YOU have to realize is that you (like many others in the past) can only seem to understand the explanation for something in terms of only what is already understood. And that there is nothing "magical" or "special" about our current understanding (unless you believe there's nothing new to discover, which is preposterous hubris).