Nuclear has amazing power ratios but detonating a nuclear warhead to get moving is somewhat of a problem when you're close to earth.
Fission Heat Engines are quite good but not good enough for takeoff. Ion engines suffer the same fate.
Almost no alternative propulsion method can replace takeoff boosters, thusly limiting the mass you can put into space. With limited mass you can only make engines that big. Or bigger if you assemble them in space which opens other cans of worms.