Aluminum isn't ferromagnetic so magnets don't work with it and stripping the graphics off a soda can alone requires just as much energy and nasty chemicals as other recycled materials. That's before you even smelt anything, which usually requires a forge that spends insane amounts of energy just to keep it from shutting down and needing an even more expensive cold start.

It's not that recycling aluminum is easier than other materials, it's that making aluminum from raw bauxite is like the above process times ten. The chemicals are even more dangerous, the process management (environmental protection, regulatory compliance, etc) more expensive, and it requires even more energy. As a result, between 1/3 to 2/3 of all aluminum sold in the US comes from recycled materials because of economic reasons.

Most beverage cans are made from aluminum, which is not magnetic.

I assume the main difference is the high temperatures used to melt metals like aluminum back down to ingots will either burn off all the contaminants or bring their remains to the surface to be easily removed like skimming dross.

Plastics recycling is a much more delicate process, requiring all sorts of washing steps and finely controlled relatively low temperatures.