Lithium is very poisonous and dangerous to recycle. It’s also poorly regulated. Solar panels have life spans of ~30 years and presently they mostly go into landfills without anyone recovering the materials that may be dangerous in the environment.
That doesn’t really matter much though. Uranium isn’t dangerous by itself, and has the highest energy density of any existing source.