I'm deeply skeptical of AI doomer stuff, but I've been struggling to exactly articulate why. I can think of it in my brain and can try to explain but so far none of my explanations have seemed fully coherent to me.

I studied biology in college. I actually did this because I was interested in AI, and already knew a ton of CS and programming. I was already coding in C and using Linux and writing fairly complex stuff. I figured studying the only actual example of a truly intelligent system (especially at the time, early 2000s) was the best way to understand intelligence.

Didn't end up going into AI as my day job, but I've kept up with it all and worked through the math and played with different types (genetic, neural nets, transformers/LLMs, CA / alife, etc.).

I think the root of my skepticism is in what I learned studying biology about how life, adaptation, learning, and evolution actually work in the real world.

I'll try to TL;DR it: I think the doom crowd (and also some of the AI singularity crowd) has a "mystical" view of intelligence. Intelligence is magic pixie dust, and intelligence beyond human is fully unknown and likely has magic superpowers.

The deeper underlying thing seems to be some kind of mathematical mysticism.

Real intelligence is ubiquitous in nature. Trees, grass, bugs, and even bacteria are intelligent. It's also domain specific and limited in capability to what its domain, available training data, and available compute allows.

It's not context-free limit-free magic. It's not special. It's an adaptation like any other.