This is a problem that I hope attracts much more scrutiny in the near future, and beyond the domain of what's called "AI" right now. It doesn't feel talked about much at all. It feels like we can analyze the inner workings of new technology endlessly, because shiny new tech is cool and racks our brains, but hardly ever stop to consider
why we keep pressing on to make even more advances. "Because we can" isn't a satisfactory answer, especially not when the stakes are so high. Why do we keep saying that? Can we ever be totally satisfied with what we already have available to use? Does this imply there will never be a stopping point? Would we go
beyond inventing AGI if such a thing were possible?
I think the idea of resistance to change to preserve our chances of survival is worthy of study. I hope there will be serious anthropological studies investigating the nature of that kind of resistance and the opposite, driving insatiability of unbounded, overextended curiosity that marches forward no matter the cost - even if the chances of it affecting the final outcome are slim. It doesn't seem like there's much of an alternative besides repeating "pandora's box, cat out of the bag, genie out of the bottle" as a post-hoc justification for every set of unintended consequences for the rest of time.
The fact that we have imaginations and can foresee the potential consequences of actions has helped our species for so long up to this point. But if safety regulations needing to be written in blood to be effective is really the way our species is most likely to operate, then that has terrifying implications. I think 10 years from now a television crew will sit down the original researchers and engineers behind this tech branded AI and ask: Why didn't you stop?