You should be more worried about those elements washing into the soil. But it turns out that zinc is an essential micronutrient (as is iron) and aluminum is a large part of clay soils anyway.
Other components of solar modules, roughly in order of mass:-
- Glass. This is heavily purified and baked soil. Not a problem.
- Silicon. This is the reduced metallic form of dirt, the equivalent of steel compared to rust or iron ore. Rusted silicon is all around anyway.
- Aluminum wires and connectors. Already covered.
- Plastics used to insulate wiring. Here we might have a problem, with microparticles wearing off and entering our food. However, the effects of this are not at all obvious - people are not dying in millions from agricultural plastics, used for example on the ground as weed suppression around strawberry plants. More research is needed.
By comparison with other uses of plastic in agriculture, the scale of this is tiny.
- Copper internal wiring and connectors. Copper has been used for centuries as a crop spray, and it is permitted in nearly all "organic"/"green" farming certifications.
- Silver, in tiny amounts as top conductors on panels. Has antimicrobial properties in certain chemical formulations, but note the "tiny". Almost certainly benign, and if there is enough of it in the soil (well below any possibly toxic level) it will be recovered for re-use.
- Trace amounts of "exotic" elements in the silicon. If these make their way into the soil, it will be in concentrations of parts per trillion, likely far below background levels already in the soil. (Some of these are things like phosphorus which is an essential nutrient anyway.)