Nuclear fuel cannot be recycled. Nuclear fuel is consumed during use, that's why it's fuel.

A spent solar panel contains the exact materials needed to make a new solar panel. Chemical changes are cheaply reversible, so you can turn an old solar panel into a new one for less energy than it will produce during it's lifetime. A solar panel factory can operate indefinitely. (Or at least for as long as their is solar energy available.)

Nuclear fuel must be mined. Nuclear changes are reversible, but that requires the same amount of energy you would get back out. The Earth has a finite amount of nuclear fuel. (To be fair, there's a lot. The Earth contains enough fuel to last roughly the lifespan of the Sun.)

This supports your point even more!