To unpack your point a little bit: what does it mean for "me" to be obsolete? Let's take that at face value and engage with it. I consider myself to be a decent SWE, maybe top 5% or so. There are somewhere around 40M SWEs in the world. If I am obsolete, that means AI is doing the work of ~38M people. I don't exactly understand what you mean by obsolete. In this future where AI is doing the work of 38M people, is anyone watching over the AI? How many people? Does it scale with the 40M figure, and if so, does it scale roughly O(n) or O(1)?

I think you basically have to accept there will be one of two options -- either it scales at something like O(n), in which case you have some number of highly skilled professional which can use AI to do the work of 100x of a SWE today (that would be a pretty cool job). Or it scales with something like O(1). I think in the second world the only logical conclusion is that AI is so incredibly capable that the world is profoundly and utterly transformed. If AI can do the work of any SWE without oversight, it can basically do anything -- which would mean that any need you have could be satisfied. If you are hungry, food could be grown. If you are thirsty, water could be procured. Or you would need to propose some crazy future where AGI is achieved but for some reason trivial things like this can't be solved.

I think the thing that makes me the most nervous is some enemy actor going rogue with AI and killing us all, but somehow we've lived for decades with enemy actors having access to nuclear and bioweapons and somehow we are all still here.