On Endianness (2021)
technicalsourcery.net
technicalsourcery.net
The OP's method only works under the following assumptions: (1) length in bytes is power of 2 so the re-interpret trick works, and (2) the complete source data is in memory.
In many popular formats (MKV, WebM, etc) the header is variable-length as well e.g. 1-8 bits, the complete integer is therefore [ 1 .. 8 ] bytes, and because these files can be very large they don't fit in memory, need to stream from disk.
Under these conditions, the big-endian format (a) eliminates need for any bit shifts, only need to mask away the header in higher bits, slightly faster on most CPUs (b) preserves original bytes in the number except the MSB which contain the header, helps with debugging.
I think that's the only use case where big endian has a substantial advantage. Fortunately, all modern CPUs have fast instructions to flip the endianness, e.g. on Intel/AMD they are bswap for 4 or 8 bytes, ror/rol for 2 bytes, available as intrinsic or standard library functions in most programming languages.
For all other cases, including binary formats I design, I only use little-endian convention. Little-endian CPUs have won, and I don't like doing unnecessary work flipping these bytes. I'm lazy, also a boilerplate like that is a common source of bugs.
My brain working set often loses things like proper list enumeration. :-)
Or when talking about bignums:
> Although this scheme can be realized with either byte order, there is an extra advantage to little endian byte ordering: If the CPU is little endian, you wouldn’t even need to care about the element size in the array because the bytes would naturally arrange themselves smoothly in little endian order across the entire array.
But the same is true for big-endian! The array indices will run the opposite way, but so what?
This seems aimed at forcing the conclusion that little-endian is better. There's no real advantage to one or the other. The world just has both, and we have to deal with it.
Think about who used numbers back when this decision was made - small business people who mostly did math by hand - addition in particular - which we do from small digits to large - if we'd done the sane thing when we borrowed arabic numbers we'd write them in the order that numbers come out of an addition operation, rather than having to guess at how much space to leave for the result and write them backwards from the order we normally write them
I don't know about you, but if I am trying to reason about hexadecimal numbers then I just separate it into 0xDEADBEEF = D*16^7 + E*16^6 + A*16^5 + D*16^4 + B*16^3 + E*16^2 + E*16^1 + F*16^0. The endianness only changes the order of the bytes I start reading at. What we ought to do is make a new prefix for reading the hexadecimal numbers in little-endian order like 0xDEADBEEF = 0yEFBEEDDE. Of course, this doesn't really fix the problem (wanting to read the number with smaller symbols first) as bytes are still considered to be in big endian if you consider the semantics about left/right shifts, which play on our preconceived notions of big endianness in everyday decimal math. You would want a system where everything is treated little-endian (bits within bytes, bytes within arrays)
Your point about bit shifts is good, too. If the rightmost bit is the least-significant bit, then for consistency perhaps memory should be written with the lowest memory address on the right. Then 0xDEADBEEF makes sense both in terms of bytes within the value and bits within the byte; the first 4 bits within the lowest byte in memory are F, then next 4 bits within the lowest byte are E, the first four bytes within the second-to-lowest byte are E, the next 4 bits within the second-to-lowest byte are B, etc. So if 0xDEADBEEF were stored in little-endian, a hex editor could display DE AD BE EF
In LE, bit n has value 2^n and byte n has value 256^n. In BE, bit n has value 2^(k-n) and byte n has value 256^(k-n) where k is the maximum length; it causes increase of ordinal position to not correspond with increase in value, and makes it length-dependent.
Bit numbering does not necessarily follow byte numbering. Personally I favor BE byte-order—if only because it means a standard hex dump shows the bytes in normal reading order regardless of grouping, which is especially helpful when larger integers are not naturally aligned, and it also matches the order of digits in (English) text strings—but I would agree that bit 0 should always be the least-significant bit.
- list memory right-to-left so the lowest memory address is on the right and the highest is on the left (i guess i prefer this solution because it's compatible with the convention of the least-significant-bit being the rightmost bit, which is enshrined in the name of "right shift", and with the convention of the least-significant-bit being bit 0, which makes sense in formulas like "value = sum_digitindex digit[digitindex]*radix^digitindex")
or you could
- write hexadecimal values with the most-significant-digit on the right, so the byte 2 followed by the byte 10 (in decimal notation) would be written 20 A0 instead of 02 0A.
I don't know if popular hex dump programs have switches for either of those, though?
I can't say I've encountered any hex editors which support either option.
Also network order is big endian because most of the machines back then were big endian, most notably the PDP-10 which was the common research machine in that era. Big endian offered simplicity in implementation (remember the earlier models of these machines were mostly hand made, even if made in a factory, with only a few semiconductors; the ALUs and instruction decoding was all done with wires, not traces). Bytes weren't necessarily 8 bits wide and while it was trivial (easier than in C actually) to do pointer arithmetic, a "pointer cast" is a weird way to think of it.
So the article is full of the assumption that the world is basically a PDP-11. I think the obsession with such machines has held computing back as much as it has sped it up.
As for the technical reason, consider wiring up the arithmetic unit by hand (work it out on a piece of paper). If you don't have an architecture book handy, look at Ken Sheriff's walkthrough of the Z80 die and consider some of the extra complexity (less of an issue of course for an MPU that was an IC) to handle little endian arithmetic ops (routing carry and such).
Come to think of it though, byte pointers move right in the word as they're incremented. Guess you could count that as big endian.
[Various replies (and votes) indicate I had better edit to clarify:] How come the single byte value written as 0x12 means "eighteen" and not "thirty-three"? Shouldn't the two four-bit nibbles also be considered as little-endian? Or it could even mean "seventy-two", if the bits are little-endian as a whole?
But most of us are used to getting them a byte at a time through the memory interface so we are ignorant of this.
They do if transmitted over a serial port. But today's networks encode whole bytes or more into the symbols actually sent over the wire/air, so there's not really any way to point out on an oscilloscope trace where particular bits out of a byte are sent in a specific order.
Because if it's just that the name "left shift" corresponds to an operation that multiplies by two and vice-versa, you could probably just swap the names "left/right shift" and be none the wiser.
Edit: the point I'm making is that you need to use either a left/right shift to extract individual bits from a byte B, or calculate some AND(B, L) where L is some literal like 0x4 (in which case it depends on how you write the literal's bit order).
Succinctly, why is the big-endian number 0x01234567 not known as 0x76543210 to little-endians? Why are they OK in calling it 0x67452301? That sure looks like a mix of big-endian nibbles in the little-endian bytes.
That means it has already interpreted it as one quantity, so at that point endianness is completely irrelevant.
Succinctly, why is the big-endian number 0x01234567 not known as 0x76543210 to little-endians?
Once again you are mixing up the visual representation with the in-memory. There is no such thing as "the big-endian number" or "the little-endian number". You have already written an indivisible number, and interpreted it according to the human convention.
How about this way: The decimal number 19088743 is stored in a 32-bit memory word on a big-endian machine. When asked what the four consecutive bytes are, from low address to high, the debugger displays "0x01, 0x23, 0x45, 0x67".
Now, let's put that same decimal number into a 32-bit memory word on a little-endian machine. When asked what the four consecutive bytes are, why does the debugger not display "0x76, 0x54, 0x32, 0x10"? That's what would happen on a truly little-endian hardware system, as used by little-endian human thinkers, expecting a little-endian human display convention. Instead, it displays the weird hybrid "0x67, 0x45, 0x23, 0x01" version. Why aren't the humans who like little-endian not bothered by the fact that the human convention is not completely little-endian, and shows the two hex digits within each byte with a big-endian human display convention?
Or, how about this: We all agree that decimal numbers as currently written by humans follow big-endian conventions. Certainly someone could come along and claim that they preferred little-endian representation for decimal numbers, so that one thousand two hundred thirty four should be shown as 4321 rather than 1234. Great; good luck to them. But if they said that they're actually going to write it as 3412, I hope we'd agree that they weren't really following little-endian conventions for displaying values for human consumption.
So, yeah, I'm complaining that the current widely accepted human convention for little-endian display is not self-consistent and not fully little-endian. And, you're right, it has nothing to do with the actual hardware, which doesn't care how we display stuff.
Because from a hardware architecture perspective, a single byte is a single digit. The order of the bits within that digit has no bearing upon the programmer. A debugger just converts the bits to a human readable format, be it decimal, hexadecimal or whatever. If you ask GDB nicely, it can print them out as decimal for you, through the "x/4db" command. You can also get it to show you the entire thing in big endian hex through "x/1xw". You're essentially complaining about the single-digit view using multiple digits for human readability.
Why don't people who claim to prefer little-endianness actually follow through and display these two ascii characters in a little-endian order? Why do people who like little-endian not prefer the human readable hexadecimal display of eighteen to be 0x21 rather than 0x12? If these humans really prefer little-endian human readable hexadecimal display of multi-byte values, then why don't they also prefer little-endian human readable hexadecimal display of the two four-bit nibbles in single-byte values?
How is it consistent that little endian people say "0x0A0B" means "eleven times two to the eighth power, plus ten"; but they don't say that "0xAB" means "eleven times two to the fourth power, plus ten"?
In what world does it make sense that the ordering of the ascii characters that represent each 4-bit nibble of a multi-byte value hops back and forth between being less significant and more significant as you read them left to right?
So, before then we didn't have a common way of describing this I guess. It's crazy to me if true.
Good paper BTW. Worth reading.
The root of the conflict lies much deeper than that. It is the question
of which bit should travel first, the bit from the little end of the
word, or the bit from the big end of the word? The followers of the
former approach are called the Little-Endians, and the followers of the
latter are called the Big-Endians. The details of the holy war between
the Little-Endians and the Big-Endians are documented in [6] and
described, in brief, in the Appendix. I recommend that you read it at
this point.
It's hard for me to call either of those advantages in anything but the most technical sense, but it's there.
(arr[x/8] >> (x%8)) & 12. x86, ARMv8 and RISC-V are Little Endian.
3. IBM 360 through s390x are Big Endian.
4. MIPS, PowerPC and ARM can be bysexual.
[edit] even ARMv8, the instruction fetch is always treated as little endian however the data access is implementation defined and therefore can also be bi-endian. [1]
[1] https://developer.arm.com/documentation/102376/0100/Alignmen...
But it does seem odd that they give "convention" to big endian despite the fact that basically all modern processors are little endian and there's absolutely no advantage to following "network byte order" if you don't have to.
[-] https://www.rfc-editor.org/ien/ien137.txt dates from APRIL the FIRST 42 years ago?
Could/would/should it matter?