Endiannes and Polymorphism in Rust | Hacker News Reader

Endiannes and Polymorphism in Rust

thecodedmessage.com

38 pointsloige28 comments

I’m writing my own programming language that deals with this in a different way. You have your variables, and you have qualifiers like const or mutable. There’s also storage order qualifiers. When you say “bigend i32” that means a 32 bit integer which will be laid out as big endian in memory. But you can still use it normally, to the programmer it has no weird inverted value.

https://gitlab.com/xmdr/ananke/-/blob/master/drafts/example....

So far, progress has been a bit slow. I still don’t have a working interpreter, not to mention compiler, because I find myself tweaking the syntax a lot. Perhaps later in december it will have finalized enough for me to dare write it out.

I agree with this. Having it declared once and letting the compiler handle it would be more convenient and invite fewer errors than requiring the programmer to insert a conversion every time the variable needs to be read or written.

Endianness in bitfields is not straightforward though.

you're right, and I'm still struggling with how to do this syntactically in a way that makes sense.

If you're looking for prior art, Ada has the Bit_Order and Scalar_Storage_Order attributes[0]; the latter is a GNAT extension.

[0] https://www.adacore.com/gems/gem-140-bridging-the-endianness...

If you're gong to the trouble to use an overlay structure to represent the serialization of an object, then the most optimal thing to do is to perform the endianness conversion once at the boundary between the outside world and the program.

Adding type annotations encourages a coding style that makes many conversions at every point of use, instead of only once at the boundary.

Endianness within bytes (or words) is actually a thing

- IBM long numbered bits from the most significant starting with 0 (think of 360s and the like, not PCs)

- there are modern chips with registers numbered the same way (a current family of multi-core ARM chips I'm currently working with number their GPIOs this way too)

Given that most CPUs are or can run in little-endian mode these days, why are so many new protocols and file formats using big-endian for storing numbers, like record lengths etc?

'Network endian'. Not saying it's a great idea though.

Sure, but why? For something that's on the application layer, why should network byte order dictate things?

Why should the ubiquity of a particular hardware design dictate that you diverge from the canonical representation of a number? If we were laying out digits and not bytes, everyone would find it bizarre to lay it out little endian, we all know the canonical layout of the digits of a number and changing that really requires some documentation. Why would changing radix change what is canonical? It gets really annoying when you convert from radix 16 to radix 2 and suddenly your "little endian" layout is actually mixed endian because rarely will people apply the endianness to representations of bits (of course, you already see mixed endianness in radix 16, it is only in radix 256 that little endianness exists and isn't actually mixed endian).

So when people are choosing to write things to wire, big endian seems quite natural because what is natural in radix 10 is equally natural in radix 16 and equally natural in radix 2.

> Why should the ubiquity of a particular hardware design dictate that you diverge from the canonical representation of a number?

Performance. Byte-swapping "everything" isn't free. And there's already precedence for it, after all most binary formats use multiples of 8 bits for the same reason when storing numbers etc.

Because people are lazy in general. It’s easier to follow what perceived to be a convention, without questioning these conventions. Programming books and university classes don’t update that frequently, the amount of old information creates that perception.

I am only using big endian for variable-length integers: https://www.rfc-editor.org/rfc/rfc8794.html#name-variable-si...

In that particular use case, allows to find length of the integer by only reading the first byte. This feature helps for use cases when you wanna skip fields instead of de-serializing them.

And, it doesn’t screw up bit patterns. A variable-length integer 0x808080 gonna have three 0x80 bytes somewhere in the serialized stream. This feature sometimes helps debugging things. A little endian equivalent would have them bit-shifted by the size of header.

For fixed-length integers, I only using little-endian in my file formats.

The natural endianness is "little endian", as it was originally devised in India. We just do it backwards (big endian) in left-to-right writing systems due to inertia (though originally for backwards compatibility with existing mathematical texts in the 10th century).

Little endian has too many advantages to be ignored, which is why all popular architectures and also newer protocols are now little endian.

I did a write-up about it here: https://www.technicalsourcery.net/posts/on-endianness/

Shouldn't your discussion on "natural order" consider the order numbers are spoken in? Which is mostly big endian.

Yes, but that is also a cultural artifact. German, for instance, still contains vestiges of the little endian spoken order, even though their adopted writing system is left-to-right and their adopted written numbering system big endian.

This is a good explanation, but I still find this a difficult concept to wrap my head around. I think the main source of confusion is that endianness (usually) concerns bytes, not words. Why is this the case? ALUs work on words, not bytes, right?

Other way round. Endianness applies to numbers/words not bytes. A byte is a byte is a byte. There’s no endianness. You read a byte off the wire, the ALU will work with it correctly. You read 2 bytes off a wire and call that a short, well which byte contains the MSB? If you dumped those 2 bytes from a little endian system, the second byte contains the MSB. If you dump those two bytes from a big endian system, the first byte contains the MSB. You only have to care if the other system is a different endianness natively or serialized to a concrete endianness.

The biggest mistake these days is protocols that use big endian as the serialization order even though most data gets processed on little endian systems.

You're correct, but I was unclear. What I find confusing is that there are two concepts in ordering of bits, and they are not "in sync": bytes and words and the order of individual bits. This means there is both "least significant bit" and "least significant byte".

# Little Endian Machine 87654321 87 65 43 21

# Big Endian Machine 87654321 21 43 65 87

# Big Endian Machine, word-wise 87654321 12 34 56 78

In a world where endianness concerns only words, there would only be 12345678 and 87654321, but not the word-wise mixing as in 21 43 65 87. I suppose this is because the individual adressable unit of memory is a byte, and byte streams can be interpreted in different ways (as int8, int16 etc)? And because bytes are usually opaque to the programmer into how they lay out their bits internally to represent a uint8?

It makes sense, but without a lot of context around it, it is very confusing initially.

The bit order within a byte can be thought of as always big endian. In fact you can think of it this way for any multi-byte number (floating point too). The reason is that the CPU takes care of this all for you (because sanity). So what the little endian CPU is doing is saying “read 16 bit 0xcd 0xab from main memory” which results in it storing the number as “0xabcd” in the register. If you said “read 32 bits from 0xcd 0xab 0x00 0x00” from memory the CPU would see “0x0000abcd”

This is all internally consistent. However, if you said on your big endian machine “write the 16 bit number 0xabcd to main memory” it would store “0xab 0xcd”. Then you say “write those 16 bits to I/O and OS I/O routines don’t understand the structure you’ve assigned to memory so they just copy that to the wire. This leaves you with “0xab 0xcd” on the wire. Then your little endian machine says “read 16 bytes from the wire” and gets “0xab 0xcd”. But then you’re smart and you go “I was expecting a 16bit number here - cpu read this as a 16 bit number” and the CPU goes “main memory stores little endian 16bit numbers so I see 16 bit number 0xcdab” (because 0xcd is the least significant byte”.

The smallest addressable unit of CPUs you encounter is bytes whatever size the CPU vendor chooses that is in terms of bits. So a CPU would never read a little endian or big endian byte because it’s simpler to just pump the bits directly instead of swizzling them (easier to debug, analyze etc).

You could design a CPU that stored numbers in bit-order little endian or big endian but you’d basically be incompatible with all software out there and no language provides support for dealing with that because you’d be very niche.

It’s because memory is generally byte addressable. The byte is the smallest chunk of data CPUs can index from memory. So endianness comes from how you order the bytes when accessing memory.

When dealing with low level hardware protocols bit endianness does become a thing.

If I ever got a tattoo, I would get an explanation of endianness on my left arm because I always mix them up

Sure the concept is easy, but have you ever met someone who mixes left and right up? It can be quite catastrophic in traffic

My problem with "endian" is the overloaded definition of the word "end" (does a 1-dimensional object have 2 ends or 1 start and 1 end?), and the ambiguity about whether you're talking about the "end" of the number or the "end" of the segment of memory. If you take "end" to mean "the end of the memory segment" (as opposed to the start) then the two definitions are the wrong way round. I think something like "big-first" and "little-first" would have been much more intuitive.

The interpretation that I find the most intuitive puts them the wrong way round, to the point where I remember which is which by specifically remembering "it's the opposite of what you'd think".

And at that point I usually look it up in wikipedia or on my left arm, because I'm not sure anymore

Every time I try to use a “it’s the opposite of what you think” mental shorthand, I get used to the wrong-seeming (but correct) way it is, and then I start mentally flipping it back to the actually incorrect version.

> have you ever met someone who mixes left and right up?

There's no right, there's only left and the other left.

I've always liked "left and metric left".

If this was HN front page bingo, we'd have a winner.

Reply on news.ycombinator.com