Challenge: specify a mainstream architecture (e.g. targeted by Debian stable) which doesn't have char=1, short=2, int=4, long long=8 bytes with GCC or Clang.
https://wiki.debian.org/ArchitectureSpecificsMemo
It shows that there are no such Debian architectures.
Also floating point formats can vary a bit even with the size being the same. So a piece of code written for one might be subtly wrong for another.
And why would you not be able to treat one as an integer? Are there any examples of such ISAs?
Finally, floating point formats do differ, but that shouldn't affect anything except precision and binary format. Also I'm pretty sure "float" is standard IEEE 32 bit and "double" is standard 64 bit, with predefined exponent/mantissa sizes. The only thing that could differ between arches is endianness, and by far most ISAs have settled on little endian.
It is hardly legacy yet.
Separately, there is also the x32 ABI, although that hasn't really gained any traction.