Some phosphate rich rocks might be also known as.. fertiliser.
Arsenous rocks would be a disaster. It's a huge issue in India with water table and ground contamination. Likewise lead, or salt (in the wider sense if it causes brackish ground water) although that's about deep wells. I have no idea if this is a serious risk for rockdust plans.
For enough upside benefit downsides might have to be borne or mitigated.
There are lots of opportunities for harm, but plenty of material that would be beneficial, particularly for crops that like higher pH.
Olivine, e g., often has excess nickel, which you then would not use on farmland. You might dump that on beaches, instead. The main thing is to move a lot of it. You would prefer to use renewable electric to crush it, but the main thing is to get started.