I assume the main difference is the high temperatures used to melt metals like aluminum back down to ingots will either burn off all the contaminants or bring their remains to the surface to be easily removed like skimming dross.

Plastics recycling is a much more delicate process, requiring all sorts of washing steps and finely controlled relatively low temperatures.