Not my area but I think the fuel is the problem, not the power source. Even a fusion-powered drive needs to spit something out the back to accelerate (e.g. ions), so you need to carry many kg of ionizable fuel.