Comment by amluto | Hacker News Reader

Comment by amluto | Hacker News Reader

Fortunately this is a bit less relevant today as Windows loses market share in database and server applications, but:

UUIDs have historically massively screwed up endian handling. While this new draft discusses sorting UUIDs as strings of octets (bytes) and the text of RFC4122 is fairly explicit about most significant bytes coming first, the C UUID structure in RFC 4122 appendix A is entirely misguided:

    typedef struct {
        unsigned32  time_low;
        unsigned16  time_mid;
        unsigned16  time_hi_and_version;
        unsigned8   clock_seq_hi_and_reserved;
        unsigned8   clock_seq_low;
        byte        node[6];
    } uuid_t;

Those aren’t bytes — they’re integers of various sizes. (Hint: do not use integer types in C code for portable data structures. ntohl, etc are a mess. Just use arrays of bytes.)

I don’t know the whole history, but MS somehow took this structure at face value and caused problems like this:

https://github.com/uuid-rs/uuid/issues/277

So, if you want to do anything (e.g. sorting) that depends on the representation of a UUID (or even depends on converting between string and binary representations), be aware that UUIDs coming from Windows may be little-endian. In my book, this is a Windows bug, but opinions may differ here.

UUIDs (as GUIDs) in Windows predate RFC 4122, so I don't think it's that unreasonable that they're not compliant with the spec (since the exact contents of a UUID aren't typically that important, but consistency in how you produce them is).

The GUID implementation in Windows derives from the DCE RPC specification [1]. That's where the multibyte integers in the RFC 4122 specification come from (they're stated the same way in the DCE RPC spec), and it doesn't explicitly specify their endianness. It does call them "NDR integers", but NDR integers can be little endian or big endian depending on implementation. DCE specifies a mechanism by which you'd indicate which you're using in an RPC call, but that data's not included in the UUID format-- you get whatever byte order the system's decided to use, which, for Windows, is little-endian.

[1] https://pubs.opengroup.org/onlinepubs/9629399/apdxa.htm#tagc...

It's actually worse than that. The first 3 groupings (textually) of the uuid might be little endian while the other 2 are big endian. Learning this cost me more time than I care to admit.

https://en.wikipedia.org/wiki/Universally_unique_identifier#...

What the…?

This fact will haunt me in my dreams :-p

I had to write some EFI/GPT code earlier this year and was dumbfounded to learn this. This is right up there with the mork file format.

Making mixed-endian even more haunted is quite the achievement. I congratulate whoever did this at Microsoft for their lasting contribution.

This is consistent with the misguided structure in the RFC. The first three fields (the time fields) are multibyte integers. The remainder is just bytes. The dashes in the textual representation are just there to confuse you.

Why would this matter (except when you build an index for a big database)?

Note:

> 6.8. Opacity: UUIDs SHOULD be treated as opaque values and implementations SHOULD NOT examine the bits in a UUID to whatever extent is possible.

When you translate between UUIDs in binary and in text form and communicate with other code the binary UUIDs. The other code might expect the uuids in a different encoding.

This mishap lives forth in the UUID stored in a machines DMI data, as well in GPT partition tables, which are required when EFI is used. It would be really cool if we had some replacement for EFI that would not harbour these kind of painful legacies.

On the other hand, this is an easy to implement conversion, while changing such a fundamental thing from EFI sounds pretty hard, making it not worth it.

You cannot fix this with an conversion because you do not know if your UUID is correct or needs conversion. DMI data for example has inconsistent endian-ness depending on the vendor. So if you have a UUID sticker on a new server, you still have two options which UUID the machine will send during PXE, either the printed UUID in big-endian encoding or in the microsoft mixed-endian encoding.

Use BIOS boot instead of EFI, it has less legacy to implement: PE executables, FAT file system, Win64 ABI

Isn’t endianness marked by the variant field? 1 is IETF, 2 is Microsoft, and this can be inferred from the sticker.

At my last workplace we twice got new workstations where they all had the same bogus UUID. Once it was Dell, the other I don't remember. The fix was to either manually set a new one in the BIOS setup, or install a BIOS Update. So I have my doubts that the variant field can be trusted here.

After failed attempts with the UUID due to above problems, we now do machine identification for PXE via MAC address for virtual machines (you see/set it in the hypervisor) and the DMI product_serial for physical machines (written on chassis).

Raymond Chen's post on UUID sort orders has been indispensable to me at various points: https://devblogs.microsoft.com/oldnewthing/20190426-00/?p=10...

It's fun to note that the worst possible UUID sort order in existence isn't Microsoft's fault but Sun's. They missed the "unsigned" catch to those integers in the struct and Java sorts UUIDs as signed integers. (Which is why sometimes you'll notice in for instance Android apps Guids get sorted such that that 8000... < FFFF... < 0000... < 7FFF...)

He's linked it here: https://devblogs.microsoft.com/oldnewthing/20190913-00/?p=10.... Its a good read. The LE format, especially with UUID 7 like UUID (seq UUID's), seems to be easier/faster to sort just using binary sorting on LE architectures which is most computers nowdays. Interesting to see all these tradeoffs.

> Hint: do not use integer types in C code for portable data structures. ntohl, etc are a mess. Just use arrays of bytes.

I don’t see how that helps much. If a developer forgets to call ntohl on multi-byte integer fields, I don’t trust them to correctly convert said integers to arrays of bytes, either.

If you write it the naive way it works.

    uint8_t bytes[4];
    uint32_t = bytes[0] << 24 + bytes[1] << 16 + bytes[2] << 8 + bytes[3];

The endianness is whatever you write in the indexing and will be the same across architectures.

Unfortunately, the naive way also turns out to be wrong in C. uint8_t gets promoted to a signed int when shifting, which in turn causes undefined behavior for specific input. One way of fixing this is casting to the desired type before the shift, thus avoiding surprising conversions.

On a side note, compiler warnings and sanitizers help with this kind of stuff greatly, use them if you have the option: https://godbolt.org/z/8oq9GTcze

>> uint8_t gets promoted to a signed int when shifting, which in turn causes undefined behavior for specific input. >> One way of fixing this is casting to the desired type before the shift, thus avoiding surprising conversions.

Agree. For clarity, the "specific input" in the example would be a bytes[0] value larger than 127.

This behavior is also explained here:

https://wiki.sei.cmu.edu/confluence/display/c/INT34-C.+Do+no...

I thought Windows uses GUIDs, not UUIDs

It looks like GUID is synonymous with UUID, but the name GUID also implies that it could contain that bug/feature mentioned[1]

[1] https://en.wikipedia.org/wiki/Universally_unique_identifier#...

Little endian won. I don't see the point of maintaining theoretical compatibility with big endian systems that are at best esoteric right now and soon to be extinct. Likewise, every reasonable platform aligns data fields on natural alignment these days. It's just a waste of effort to make software portable to evolutionary dead ends.

You're saying it won for this application (UUIDs)? Universally?

What is the most common remaining use of big endian?

TCP/IP might be a reasonable example. There's a reason "network byte order" and "big endian" are the same thing.

> TCP/IP might be a reasonable example.

That's a pretty significant example of widespread usage that isn't going away soon. Perhaps the GGP was only referring to UUID applications.

This isn't about compatibility with big-endian machines. This is about compatibility between different uuid libraries, potentially on different operating systems, all on little endian CPU architectures.

Every existing library follows the standard which specifies host byte order, which usually means little endian. The Rust library cited a few levels up this chain ignored that, somehow assuming big endian, and then had to correct for this mistake.

Reply on news.ycombinator.com