First, there are the issues of storage and disposal of enrichment byproducts, specifically depleted Uranium Hexaflouride ("DUJF" or UF6).
Second, there's what to do with old fuel rods. Proponents argue this is a solved problem and it can be stored for the long term in geologically stable storage facilities. This is just one political change away from becoming a huge problem.
Neither really scales up to 100x or even 10x current fission energy production.
Third, there's long-term maintenance of a plant. Arguments about the relative safety of fission point to deaths caused by coal. Coal is really the worst fossil fuel. LNG is better. But regardless of the numbers, the failure modes are just much, much worse for nuclear. 35 years later, the Chernobyl Absolute Exclusion Zone still stands at 1000 square miles. Parts of this zone will likely remain uninhabitable for centuries.
I just don't trust governments or corporations for the long term inspection, maintenance and management of fission power plants in the long term.
Lastly, the politics are against fission. This seems unlikely to change anytime soon.