A review of endianness bugs in Qt, and how they were fixed
mitya57.me
mitya57.me
uint32_t read_int32(unsigned char *b) {
return uint32_t(b[0]) |
(uint32_t(b[1]) << 8) |
(uint32_t(b[2]) << 16) |
(uint32_t(b[3]) << 24);
}
read_int32(unsigned char*):
mov eax, DWORD PTR [rdi]
ret
But if the big-endian case isn't run through CI then bugs will still creep in, where people forget to use the abstraction and just cast a buffer pointer to do a 32-bit read, or where they don't keep the distinction between 'a 32 bit value in host endianness' and 'a 32-bit value in host-independent endianness (eg network data, or pixel data)' straight and forget a conversion step. QEMU's testing on s390 and ppc64 catches a steady trickle of this kind of bug before it goes into the codebase.
(You can catch unaligned-access bugs on x86 with clang/gcc address-sanitizer, I think, but it can't help with endianness bugs.)
Your example is actually for the most favourable case because there are native 32-bit types. I've seen countless attempts to implement three-byte records via custom Uint24 classes that overload operators and abstract endianness (in various buggy ways) instead of just reading three damn bytes in a uint32_t. Supposedly in the name of readability ("I want to be explicit that this stores a 24-bit variable, not a 32-bit variable, so I'm going to copy-paste these 200 lines of either template magic or line noise, I'm not sure, but see how readable it is now? Oh yeah it's still 32-bit underneath, yeah, I'm not sure what happens if you write a 32-bit variable there, I think it breaks. This is way better than a typedef though, 'cause, uh, C++-17?)
// wrong
uint32_t num = ...;
// num is set to 0x10203040
num >> 24 == 0x10; // On little endian
num >> 24 == 0x10; // On big endian
One thing I have to do frustratingly often is to implement a bunch of functions like "read_uint32_le", it is a pity that there is no comprehensive standard.
([1] On the other hand, I miss the good old days where you could treat C like "high level assembler", a pointer was just a number that indexed memory, and you could reinterpret bytes with impunity - sometimes had to to get decent performance!)
I wonder what the arch difference is between the clangs that do it and the ones that don't, I don't know much about arm and risc-v.
Not sure about RISC-V but I would not be surprised if the same applies there.
(Some instructions insist on aligned data, like atomic read-modify-write operations and some types of SIMD insn, but x86 has a few needs-alignment special cases too. Basic "just load/store data" insns that the compiler will emit are fine.)
It's probably a case of lowest common denominator.
Unalligned access is discouraged but allowed by the spec... But it's allowed to raise an exception. But not required. Cheaper to optimise to avoid that circumstance.
But for the longest time people would basically dump the raw bytes of their structs from / to the network whenever possible, e.g.
struct my_data {
int foo, bar;
float x,y,z;
long dontdothat;
} data;
write_bytes_to_the_network((const unsigned char*) &data, sizeof(my_data));But yes, people often start that way and regret only when it is too late and the files are in the wild.
Imho a better way to handle this would be to use a macro or special function that comes with the compiler, assuming it exists of course.
If you know you are reading aligned memory and you need to do it with a single word memory access, then you need to use intNN_t* and use htonl or similar functions to byteswap. There’s no way around it.
cpu_to_le32(),
cpu_to_be32(),
be32_to_cpu(),
le32_to_cpu(),
These functions will either swap, or not, depending on the endianness of the target machine, and this decision is made at compile time, so the "no swap" case imposes no performance penalty.)
That way, the intent is clear, and the code works on big endian and little endian. So instead of what you propose, which I imagine you might use via:
x = some_value; /* say this is big endian, and we want it little endian, because we're on Intel. */
uint32_t y = read_int((char *) &x);
y = be32_to_cpu(x);
Likewise, you could write:
x = cpu_to_be32(y);
This is similar to htonl(), ntoh(), htons(), ntohs(). (host to network/network to host long/short).
Except it is. On x86, which is little endian, compilers are smart enough to convert it to a mov.
> These functions will either swap, or not, depending on the endianness of the target machine
So will the above. Here's the MIPS little endian version:
read_int32(unsigned char*):
lwl $2,3($4)
j $31
lwr $2,0($4)
read_int32(unsigned char*):
lbu $3,1($4)
lbu $2,2($4)
sll $5,$3,8
lbu $6,0($4)
sll $3,$2,16
lbu $2,3($4)
or $3,$5,$3
or $3,$3,$6
sll $2,$2,24
j $31
or $2,$3,$2
But it is. That's the whole point, to not bother with weird endian conversion macros.
You're telling me that what the linux kernel guys do is "weird endian conversion macros?" I don't believe you.
It doesn't always swap. Look at the assembly I posted above comparing mips64le to mips64be.
The code takes the byte at b[n] and shifts it (8.n)-bits. These are the same bytes on both little endian and big endian machines. They read the bytes in the same order (little endian), and convert them to the native order.
You can also see very clearly that it doesn't always swap by looking at the simple x86_64 assembly output that the was posted where no shifts or swaps are performed. Just a mov instruction.
Read the assembly. Or better yet, try the code out for yourself.
Assuming LE ordering on the writer side, you would need to swap again on a BE machine. Usually, however, you want to store in your target byte order, though. One could argue this is irrelevant as long as the memory doesn't leave your application, of course. Yet, by explicitly having macros _to_cpu, _to_be, _to_le and so on you make all of this explicit.
We will still have to deal with the legacy network protocols that have big endian encoding.
By the way, if you are designing a new network binary protocol, please do not pick big endian, it just makes stuff more complicated on the majority of current cpus.
The only reason that big/mixed endian is even an issue is because some monks fucked up back when the Moors were driven from Spain ... in the wonderful libraries that were left behind they discovered Arabic numbers and realised how useful they were, far better than the clunky Roman ones - for a start a small businessman could do addition just right there on the page, writing the digits from smallest to largest as they went - sadly as I said the monks screwed up and copied the digit order from the right to left written Arabic into western languages without doing the sane thing and reversing the digits so that they continued to be written smallest to largest (so that you could do addition without knowing how much space you needed for digits until you were done).
And so we're stuck with this silly artifact of history that has caused decades of arguing as we slowly rediscover the utility of putting the smaller digits first, as they were always meant to be
In the Unicode characters should be ordered logically (in the order they are interpreted) not visually. Then you use RLM or ALM to indicate right-to-left.
Some IBM systems do number the MSB bits of bytes as bit 0 - to confuse things they are nominally big/mixed endian
The long term good news is that the endian wars are drawing to a close. Little Endian won. I believe all the most recent CPU's support little endian at least as an option.
From what I know from talking to IBM engineers, there are no plans to switch AIX on POWER or Linux on s390x to little-endian.
And both are important platforms in the enterprise world.
They’re not going anywhere.
I know they're not going anywhere, but they're also largely irrelevant to majority of us.
We've seen POWER largely migrate to little-endian, as IBM was struggling with sales due to the difficulty in porting software to it, partly due to the endianness difference. (Note that POWER and PowerPC were technically always bi-endian architectures, but the majority of historic hardware had its endianness set by the motherboard in use, and almost everything shipped with big-endian. Systems running Windows NT on PowerPC were some of the very few that ran in little-endian mode.)
We've also seen a similar shift in embedded hardware; bi-endian architectures like ARM and MIPS are now more commonly used in little-endian mode, despite historically being used in big-endian mode.
Little endian also allows you to truncate integers more easily.
In the days of 8-bit microprocessors, the endianness became visible in software from 16-bit numbers up.
Now it might be visible only for numbers above 64-bit on many CPUs, but there is always a number size above which you must take endianness into account, even for the simplest operations, like comparing and adding.
In theory, you could choose the layout of a large number to have mixed endianness, e.g. to have a large number as a little-endian sequence of 64-bit chunks, where the 64-bit chunks are big-endian, because the CPU used is BE.
However that might make the algorithms even more complex than when you use a consistent endianness.
The endianness of a CPU is not visible in software only when the addressing unit of the memory is the same as the data register size of the CPU.
All modern CPUs have register sizes of either 32-bit or 64-bit, while the memory is addressed in bytes, so the endianness will continue to matter in the foreseeable future.
Some operations start at the least significant end, some start at the most significant end, and it's equally easy to loop over the words in either direction, so i don't buy this.
> Little endian also allows you to truncate integers more easily.
What do you mean by truncate an integer? If you mean go from 32 bit to 16 bit, that's still trivial with big-endian, you just shift.
Surely you would agree that moves are easier than shifts?
Moves can be handled by the renamer and be effectively free (even for memory operands, on CISC, with the advent of memory renamers; but this is even more important for RISC, where what would otherwise have been a load followed by a shift can just be a load), whereas shifts cannot and must take up some fraction of a cycle.
It's the same kind of thing as computers and calculators switching from decimal hardware to binary hardware 100 years ago.
Not quite. ENIAC, for example, did calculations in decimal, and it was finished in 1945. There may be more recent examples.
If you include binary-coded decimal, that still includes machines made in 1970 or later, like the IBM 1401.
Most mainstream protocols don't really take advantage of this because it is error prone, but it is used sometimes.
If you could tell cryptographers this I would be very grateful.
In the article, many of the errors were a result of undefined behaviour that just happened to work on the CI machines upstream Qt ran. Relying on undefined behaviour in your code will bite you in the bum sooner or later, regardless of endianness.
Does anyone know about some reasonably priced big-endian linux hardware that could be added to build system to run CI tests on big-endian?
The only thing i found is Ubiquiti Edgerouter Pro (MIPS64 BE).
Appears nobody has done the same for Linux yet. The link above mentions that Rpi4 isn't done because BE mode with ACPI either isn't possible, or is difficult.