A chunk of uranium-238 contains a lot of energy but it is only fissionable but not fissile, so you need an external fast neutron source to extract that energy. Comparatively it is a fairly safe energy source for its density. Similarly there are some metastable nuclear isomers that can theoretically be stimulated to release their energy with high energy photons with a precise frequency and would otherwise release their energy relatively slowly through their natural decay modes.
And I think the difficulty of turning a hypothetical portable fusion power plant into an explosive device should be evident from the difficulties of keeping current experiments running for even a few minutes.
Any nuclear scientists here?
As for weaponizing batteries, pretty sure bad actors have been able to cook dangerous bombs at home for decades that can inflict magnitudes more damage than a battery could ever do. So battery based bombs won't be a thing for the near future.