The fabrication of of panels is more analogous to fission material mining. As in you are procuring the materials that will produce energy in the future.
If we get rid of nuclear power, we don't need to mine those things anymore. If we get rid of solar panels, we still need semiconductors. So I don't think you can use it for an argument against solar manufacture.
The more semiconductors you make, the more waste chemicals you produce (and use), and the more contamination and cancer you’re going to have if those chemicals aren’t handled correctly. Aka more solar panels, more waste chemicals.
Same with nukes and nuclear waste by running your nuclear plant longer/harder.
90/10 one way will produce a lot of one thing, and less of another - and vice versa.
Why make it binary? Nuclear power plants need less semiconductors per Joule of electricity produced than solar panels.
So obviously they don't 'remove' the need for semiconductors. But they decrease it ever so slightly compared to solar power.
Just to be clear: the dangers per Joule from the whole lifecycle of both solar power and nuclear power are both really, really small.