Comment by whaaswijk | Hacker News Reader

Comment by whaaswijk | Hacker News Reader

Parent Full thread

Thanks for the substantive reply.

> How does it know it's improving and not overfitting to its own recursive definition of intelligence? It can't, and that's exactly what it will do. I haven't given this much thought, so maybe I'm missing something, but I don't see how this follows. One possible solution: to avoid overfitting, can it not just make a copy of itself, modify the copy, and empirically check if the model performs better? That's essentially what humans are currently doing when designing AIs.

Regarding the X-risk vs D-risk: I think how one weighs these risks partially depends on what one thinks the capabilities of the models are. Call me a boot-licker, but if the models get smart enough to explain, in detail, to any psychopath, how to construct a bomb or synthesize a deadly virus, I don't think benefits society to distribute them widely. Therefore, to argue for widespread distribution you have to argue that either (i) the models aren't that capable or (ii) the guardrails are robust enough to prevent them from being used in catastrophic ways by bad actors. I think we may be rapidly approaching a time where neither of these hold. Having said that, I certainly agree that the D-risk is also real.

On your twinning idea:

> can it not just make a copy of itself, modify the copy, and empirically check if the model performs better? That's essentially what humans are currently doing when designing AIs.

That's the same as setting a fixed metric for intelligence, like IQ testing, and goal seeking that.

The problem is what happens when you max that out. How do you set the next metric? Now you're back to the recursive problem of your metric "begging the question." I don't think you can get smarter by seeking "I'm smarter because I think I'm smarter and I'm right because I'm smart."

On X-risk stuff:

They can already do what you say. Try asking an ablated 30B model on your laptop how to weaponize anthrax. The answer isn’t bad. It’ll tell you how to cook meth too.

I studied undergrad biology and walked out with the knowledge to create some damn evil things if I had the right lab, time, and no conscience. This was pre AI. The recipes for a lot of nasty stuff is in open literature.

Why has nobody done this? Because… they haven’t.

That’s the answer. There's no magic stopping anyone, and it's disturbingly easy. Very rough difficulty estimate: making a novel disease (or weaponizing a current one) that could kill millions is about on par with clandestinely cooking LSD. So harder than cooking meth, but we know labs have cooked acid so it's very possible. It gets easier if you're an unhinged fanatic and don't care if you kill yourself with your own plague, which means you can skip all that bunny suit nonsense.

AI boosts bioterror risk a little, I suppose, inasmuch as it might help you learn. It doesn’t affect atomic risk much since the bottleneck there is materials. Once you have enriched weapons grade material a gun type weapon can be made in a machine shop with designs available at a college library.

As for ASI becoming sentient and exterminating us, I can’t give it zero probability because there's unknown-unknowns. But I’d rank it far lower than the risk of extreme climate tipping points (e.g. the clathrate gun), old fashioned bioterror, or atomic war. And like I said, advanced AI could be used to help us with climate change if it can help us crack better energy tech.

> That's the same as setting a fixed metric for intelligence, like IQ testing, and goal seeking that.

How does that prevent AI from becoming superhuman in all intellectual domains, thus creating ASI? We set "become good as chess" as the metric and it became superhuman in that domain.

> The problem is what happens when you max that out.

I'm not sure how you conceptualize "maxing out" the intelligence metric. Again, taking chess ability as a proxy metric for intelligence, there is no reason to believe we have maxed out chess performance, but AI is already far superior to humans. And better bots are created all the time. There is no need to design a new metric to improve chess performance. The old "How many currently existing players can I beat?" is good enough. Also, why couldn't it design better metrics after achieving superhuman intelligence?

But even if we suppose there is some kind of fixed point limit to this process, it would still be far above human level. That is all that is required for ASI.

Regarding X-risk: the point is that it becomes easy even for people who, unlike you, haven't studied biology. For things such as atomic war, the AI does not necessarily need to acquire the materials. It can access them digitally by hacking the weapons systems, possibly in collaboration with some human actors. Or maybe it spoofs detection systems causing countries to fire upon each other. Generally, it seems to me that the barrier to entry for bad actors to cause these scenarios is decreasing. Whether these scenarios are more likely than D-risk I don't know.

Reply on news.ycombinator.com