AI safety is synonymous with user hostile design. I think John Carmack genuinely cares about users.
What makes you think that?
He worked at one of the most user-hostile companies on the planet for a decade, and only left because he wasn't happy with its operating efficiency.
Someone with that mentality might make technological progress on their own, but I'm not confident that they value morals and human safety over technology. His dismissive comments about AGI safety are certainly a concern.
Imagine something that thinks 1000x faster than you, is thousands of times smarter than you, never sleeps, never eats, can self-improve, can self-replicate and it didn't like you, or someone deployed it against you and your ethnicity, family, whatever?
It would be devastating. You'd be fucked. We'd all be fucked.
Personally I don't think such a creation will ever be practical or work the way we think it will, but if we'd like to keep enjoying the world and value our autonomy, I think we should be very cautious when implementing it.
Yeah that would be bad. We're also way way off from that point. A real threat that we actually face now is someone like Sam Altman getting congress to make it illegal for individuals to run models locally because then he can't charge us subscriptions for them or harvest information about our lives, thoughts, and feelings. Imagine an AI therapist (I've already done this with llama 65B a bit) with telemetry that lets commercial interests know all your specific deepest desires insecurities and fears so they can sell you stuff.
Skynet isn't the pressing issue, a Corpo-Techno-Stasi is.
We will need AGI to ensure continued existence of the human race. Without it, our destruction is assured.
Safety here being say; a user asking for instructions on how to make a dirty bomb and the model responds with "Sorry, I can't do that ethically".
If not outright blocked, instructions do seem to be weighted down.
In that sense it is mathematically equivalent to (say) applying an L2 regularization penalty to reduce the occurrance of higher-order terms when fitting a polynomial. Strictly it will produce a worse fit on your training data, however it is done because out of sample performance is important.
The pilots need oxygen masks as a backup while they quickly descend to a safer altitude. Unless you're flying over the Himalayas and can't drop that low, the passengers should be fine.
https://en.wikipedia.org/wiki/Emergency_oxygen_system#Incide...
Given the self-evidence of no existential risk, there is then an objection to “dumbing down” where model performance suffers due to RLHF (the “alignment tax” is a real thing), and often but not always this includes an objection to wokeness or perceived unwillingness to speak “truths” or left-wing bias being imposed on models.