Elements and chemicals are very different things. Silicon is the #2 most common element on earth and if you smash a bunch of solar panels, they turn into sand when they get wet. Or if you just wait.

Even pure silicon metal is totally benign, even in large amounts. Certainly safer than iron, which can kill you in relatively small amounts if it's in a bioavailable form. Silicon is already literally everywhere, in every form it will reasonably be transformed into.

Even the boron and phosphorus that make up a tiny percent of cells are extremely benign. The biggest issues are the tin fingers and solder, and probably the lead that is often present. But lead is also far from necessary. And unfortunately, the human race is going to be dealing with lead dust for a while already, just from how it used to be in gasoline.