>unless you have some fancypants sort which understands them

FWIW GNU sort can sort by human readable units using the -h option.