I'm not a doomer, but I'm trying to understand the perspective. Up to chapter 10, I'm still not convinced. There's too much hand-waving around what will happen. It's like: 1. AI smarter than us 2. ??? 3. We're dead. What happens in 2.?
I just got past the "extinction scenario" and I'm wildly unconvinced. I agree that AI can theoretically create a bioweapon. I'm unconvinced that an AI can hide its ability to RSI, find money to run its weights on a random GPU farm, train itself to be smarter _outside_ a lab with no human input, then somehow manipulate people to give it supplies to build a bioweapon which it uses to kill us all. I to believe that Yudkowsky was good at fanfiction; this senario did not demonstrate that. My number 1 question: why do they think an RSI capable model would be first developed OUTSIDE a frontier lab? The labs have more compute, more data, more human brains working on the problem. Also thousands of variations of that same model that escaped. The escaping model somehow acquires the millions of dollars it takes to run training to somehow RSI itself into infinity then decides to kill us all, all before the frontier labs manage to achieve RSI?
Come on.
Anyway. Finishing the book for completeness, but massively underwhelmed.