Hacker News Reader

Comment by wbl | Hacker News Reader

Please explain how you send a struct over the network without UB and leaking the padding.

Use a packed struct (e.g. __attribute__((packed))). You may take a performance hit due to lack of alignment, but that's the judgement call

> performance hit due to lack of alignment

This isn't true of modern processors.

Sure you get free unaligned access for scalars on x86, but unaligned arrays are still trouble if you use SSE (which basically everyone does).

Would this work

  struct S {char data, char pad0, char pad1, char pad2};
  int main() {S dataStruct; printf("%s", dataStruct.data); return 0;}

and give me a 4-byte struct or would the compiler optimize it all away and leave me with a 1-byte struct?

Since you don't actually access the size of the structure, I see no reason why it matters. Also, FWIW, your code has undefined behavior because you call printf with the wrong type.

Yes on the UB, I'm passing a "char" into a "char *", sorry about that. I don't pass lone char data that often.

Well, I chose this example specifically to ask whether the resulting memory structure, completely independent of calling a sizeof(), would still be as if I were to call a sizeof() (which then I believe would be 4 bytes large) if not accessing the total size of the struct or any of the padding members.

It makes a difference in the memory footprint, which can become important, if you're programming a device with just 64-bytes total RAM.

__attribute__((packed)) is not part of the standard, and therefore doesn't quite answer the question. I believe the standards-compliant way of doing this is that "you can't".

By knowing its exact size with sizeof(), and using that and a cast to uint8_t* to both bzero() it out before use and memcpy()ing it when sending it. Unless the C standard says that the compiler may change the padding bytes of the existing storage?

To be clear, it is something like this, which is not bad at all to write:

  struct st s;
  bzero(&s, sizeof(s));
  // ...
  write(fd, &s, sizeof(s));

I’ve dealt with many systems that have tons of unnecessary serialization logic, and found them slower and buggier than just letting the C spec define the data format.

Having said that, it is possible to get this sort of thing right with C++ and templates (thanks to inlining).

Higher-level languages without compile time code generation (like Java and Python) usually make serialization a train wreck in comparison with “send raw structs” or “make the [C++] compiler generate and typecheck the machine code that does the serialization”

There's enough wiggle room in the standard that two compilers would be happy making two compliant but still different physical layouts for the same struct declaration. I'm not even worrying about cross-platform issues or bitfields yet.

You can get away with directly writing a struct for simple structs, but eventually it will bite you.

Packed structs are reasonably standard (even gcc and visual studio can be made to pack in the same way using the same syntax these days).

The remaining issue is endianness, but you can configure ARM to match x86 at boot so shrug.

Also, if I tell any reasonably experienced developer my protocol sends a packed struct with these fields:

  uint8_t version
  uint16_t reserved
  uint8_t op
  int32_t status
  uint32_t data_length
  // void data[]

They’ll be able to write a correct deserializer in pretty much any language, and can probably write it in ~ 10 lines of code.

I agree that packed structs can be tricky, but they’re much better than the vast majority of serialization libraries I’ve dealt with (including all of the open source ones, and I’ve used all the usual suspects).

My experience with packed structs is there really aren't any gotcha's. Either works or not.

The big annoyance is network people all use big endian probably just to spite everyone else.

Why would it be to spite everyone? Big endian is network order by definition. It's hard-coded into every asic in switches and routers for decades. If you don't use big endian your packets won't even make it to where you want. Of course, your own payload can do whatever you want as long as you know with 100% certainty nobody else will ever want to talk to your application and get confused.

Consider the lorawan spec which was written in the last 5 years. The stack is designed to be implemented in software on small 32 bit microcontrollers which are _all_ little endian. And it's big endians.

The people that wrote that spec knew that and went with big endian anyways.

Also I work with people that do network hardware. They can't care less about little or big endian. Makes no difference to them.

Someone needs to pick one or the other: you can't have it both ways.

Yeah they are picking the side everyone else abandoned 25 years ago.

> The remaining issue is endianness, but you can configure ARM to match x86 at boot so shrug.

ARM in BE mode is pretty rare / nonexistent. Small MIPS chips still float around in mostly low-end networking gear, though, and those do tend to run in BE.

A small quibble: bzero() is not defined by the C standard. memset() is.

In the "..." part you presumably meant to include writes to members of s, and those writes are allowed to change padding bytes and leak information.

Obligatory pet peeve note: sizeof is a unary operator whose argument is written in parentheses if it is the name of a type. It is not a function.

All answers to this must be wrong as it's simply not possible to do it. You need to take endianness of each field into account, and you cannot do that by treating the struct as a single blob.

Reply on news.ycombinator.com