Aluminum isn't ferromagnetic so magnets don't work with it and stripping the graphics off a soda can alone requires just as much energy and nasty chemicals as other recycled materials. That's before you even smelt anything, which usually requires a forge that spends insane amounts of energy just to keep it from shutting down and needing an even more expensive cold start.

It's not that recycling aluminum is easier than other materials, it's that making aluminum from raw bauxite is like the above process times ten. The chemicals are even more dangerous, the process management (environmental protection, regulatory compliance, etc) more expensive, and it requires even more energy. As a result, between 1/3 to 2/3 of all aluminum sold in the US comes from recycled materials because of economic reasons.