It seems like it takes just one user using a certain piece of hardware to justify it being supported in the kernel, which is surprising to me. The cost to kernel dev velocity is not weighed more heavily against that?
E.g. there's some stuff like erratum workarounds for old x86 CPUs that would be nice to drop, but they are part of a big framework for handling x86 diversity. Dropping individual workarounds doesn't let you drop the framework.
Exceptions are gonna be cases where dropping the support removed something significant from the lowest common denominator. Big stuff like word size, memory ordering (I assume dropping Alpha would be quite handy), virtual address space limitations.
You can be so flexible that it turns everything into a giant slog. I doubt 32 bit support is anywhere near that horrid; commenting on the idea more than this specific case.