If I have a "Residential scale" nuclear plant in the basement, there are lots of people who would like to re-purpose my nuclear fuel in other ways. (And that's before we talk about how I purchase, and renew, that fuel to begin with.)
Ultimately even "small scale" nuclear comes with massive bureaucratic and security overheads that can't simply be wished away.
In closing I'll point out that my solar system _is_ nuclear. It uses fusion nuclear energy as it's fuel. But I've cunningly hidden the radioactive elements far far away....