Rare earth elements actually aren't that rare. For example, neodymium makes up over 30 ppm of Earth's crust. That's more common than lead, cobalt, tin, thorium, tungsten, molybdenum, and quite a few other elements with large-scale industrial applications.

The difficult part of producing rare-earth elements is separating them from everything else. Tiny pieces of spacecraft dust scattered over a large area don't make very high grade ore...