How we're (actually) all going to die

Long before the kid asks the AI to build a black hole, national laboratories would have done the same with far more resources. Why? Because an artificial black hole is the best possible energy source. It allows direct conversion of matter into energy -- just stream atoms into the black hole and collect the Hawking radiation.

Also, I believe the energy density you would need would require focused gamma rays and very large amounts of energy. AI might be smart, but it can't break the laws of physics. It would be easier to work with visible light, but that cannot focus as tightly and would thus need orders of magnitude more energy.

Generalizing, the problem is whether an aligned AI might perform a task that inadvertently causes damage. In the thought experiment, the AI was just trying to help a kid with a science experiment. However, I think that an aligned AI that is smart enough to solve the black hole problem would also be smart enough to see the consequences. Then, if the AI is truly aligned, it would not comply with the request.