I think what's really in discussion here is what the default should be. ls was originally conceived and implemented in a period when all the files you would be dealing with could be measured in bytes easily, and would almost never be over four digits. Many common files today (any MP3, most word processing documents, almost every image except for thumbnails, etc) are nine or more digits, and at that level it's very easy to lose track of a digit or two, which is misjudging by an order of magnitude or two.

Human readable units helps quite a bit with this, and I think that makes it a sane default. As long as you can specify you want to see bytes, and the sorting is done on the actual byte value, there's little lost because the majority of the time it will help rather than hurt.