Hacker News Reader

Comment by anyfoo | Hacker News Reader

Does that also zero the padding that may exist between elements of the structure (or at its end)? Because if not, not memset'ing will open you up to all kinds of info leaks, especially if you plan to send that struct over the network (or to another process, or from kernel to userspace...).

EDIT: Actually I have to apologize, because I’m not certain what the C standard says about whether the padding bytes can change when changing members of the structure (6.2.6.1):

6 When a value is stored in an object of structure or union type, including in a member object, the bytes of the object representation that correspond to any padding bytes take unspecified values. 51)

If that means that the padding can actually also change when just assigning individual members (which I think it does, but footnote 51 only calls out copying the whole struct by assignment, and I’m not 100% sure that “including a member object” means what I think in this context), then that means that copying an unpacked structure across information boundaries is technically always unsafe, because any change of a value in the struct might leak information by changing the padding to undefined values. It’s safe with explicitly packed structs, though (which should be the majority of the cases, as you’d have a representation independent of the architecture), but then so is what the author describes. I’d appreciate if someone could clear this up! memset’ing can still help you getting a structure quickly into a well-defined state, though.

View on HN

Nope it doesn't:

  //GCC:  -O0
  //MSVC: /Od
  #include <memory.h>
  #include <stdio.h>
  struct S { unsigned char x; int y; };
  int f(int f) { S a; memset(&a, f, sizeof(a)); unsigned char y[2]; memcpy(&y, &a, sizeof(y)); return y[1]; }
  int g() { S a = { 1, 2 }; unsigned char y[2]; memcpy(&y, &a, sizeof(y)); return y[1]; }
  int main(int argc, char *argv[]) { f(0xDD); printf("%#x\n", g()); f(0xFF); printf("%#x\n", g()); }

It definitely seems like the author didn't realize this fact, or I'd have expected it to be addressed in the article.

I am aware of the fact that unitilized padding can have nasty side effects: https://lwn.net/Articles/417989/

But yeah, I should have mentioned the caveats too. I will update the post once I'm back at my computer.

I'm confused what I'm supposed to look for in that article (do you mean you were the author?) but anyway -- I feel like calling this merely a "caveat" gives the wrong impression? It seems a bit like telling people to mix bleach with vinegar and then saying "Oops, sorry, I did know that it produces chlorine! I forgot to mention that caveat." The caveat isn't just a side note for the margins, it's a critical reason why people avoid this sort of thing.

Failing to clear padding in structs is _really_ not often a problem. It's pretty much only a problem when copying un-packed structs from kernel-space to user-space.

So if you're not working on an OS kernel somewhere in the syscall path - really not a common activity even among C programmers - it's not a problem. If you are passing a struct from one part of your program to another part, or to a library you use which is not isolated or sandboxed in some way, it's not a problem. If you pass a struct to a _more_ privileged context like a syscall, it's not a problem. If you are passing structs over a network connection or writing to a file, you'll pack them and be aware of every byte anyway, in order to be compatible with another implementation reading them.

If you _are_ passing a struct from a more privileged context to a less privileged context, then yeah you have to memset().

I agree, but the frequency isn't the issue. You could say that's where my analogy breaks down, and if that's your point then maybe it does (although how often do people want to mix bleach with vinegar?), but that entirely misses the point. The point was that the failure mode, when it does come up, is a critical one, and if you're going to criticize a safe coding practice and tell people to switch to a potentially dangerous one, it behooves you to explain to them this very fact. It's just irresponsible to tell people to switch to a dangerous practice without making them aware of that fact and telling them how to deal with it.

> If you pass a struct to a _more_ privileged context like a syscall, it's not a problem.

Simply not true; that context could pass a copy of that structure to a less privileged context other than the original one, without doing anything about the padding.

(Well, you could argue that it's the privileged kernel's fault: it should meticulously floss the structure between the members to clear the padding to zero while preserving the values.)

Someone upthread mentioned "sending over a network": that's an example. Sending over a network (or local socket/pipe) begins with pushing it down to a more privileged context, which then assiduously places all the bytes into a buffer that is blasted out on the wire. In this case, the more privileged context has no idea what the structure even is; it's just a blob of bytes.

If the author is only going to add "caveat", then that sounds like word games and being unwilling to admit that the blog post's advice is wrong (for the security risks and other reasons pointed out here in the comments)

Technically I don't think memset forces the compiler to clear the padding and keep it clear either though. The magic of the C abstract machine.

Why? memset followed by memcpy must return the same byte everywhere. Otherwise what would memset even mean?

I feel like accessing the bytes that correspond to padding might be implementation-defined, but I couldn't find this in the standard.

Parent comment was edited to include this quote:

_6 When a value is stored in an object of structure or union type, including in a member object, the bytes of the object representation that correspond to any padding bytes take unspecified values. 51)_

So sure, a memcpy after a memset aught to preserve all the bytes, but if they are updated through a struct which contains padding then the values in the padding bytes will be left unspecified.

I don't think it is. It wouldn't make sense since memcpy is basically supposed to convert between arbitrary binary data.

From a practical point of view, yes, but I would be very surprised if the C standard had any accommodation for memcpy being a way to leak the value of a structure's padding.

You clearly haven't understood the thread.

How so? Did I miss something important?

Do you mean in the sense that a compiler can say "I know what memset is supposed to do" then decides that even though sizeof returns a particular size, that it might ignore that and set fewer bytes when it does a substitution for the memcpy call?

Yes. As long as the behavior observable by a conforming C program is the same, the compiler is allowed to change anything else.

Even if memset in a vacuum is guaranteed, look at the quote by anyfoo. "When a value is stored in an object of structure or union type, including in a member object, the bytes of the object representation that correspond to any padding bytes take unspecified values."

If the compiler knows you're going to write to the members right after the memset, it can put the padding back to its previous values. And by that I mean as far as your code can tell it put it back, but in actuality it never zeroed the padding to begin with.

And since there are performance benefits to pretending, now you have a situation where a "helpful" compiler and a malicious compiler have the same effect: data can get leaked.

As soon as a pointer to said memory is passed to an extern function in another translation unit, the compiler can't prove anything about how it's used, which is the case in pretty much all of the examples mentioned in this thread.

Also, type-punning is a thing. memset is byte-oriented/memory-oriented. Just because you're using it to zero a struct of a particular kind doesn't mean that's the only way the memory will accessed. Just because reading the padding of some struct is undefined behaviour doesn't mean accessing those bits by some other means is also undefined.

A compiler usually can't eliminate a call to memset() in most practical cases of initialization (where memory leakage is also a concern) because they almost always pass a reference to a routine in another translation unit. Something like memzero_explicit() can be used anyway -- but you're massively overstating the relevance of compilers eliminating dead stores done via memset(). It's much more of an issue for post-destruction memory sanitizing (which is the primary use case for memzero_explicit) than it is for compiler f*ckery when memset() is used for initialization.

> As soon as a pointer to said memory is passed to an extern function in another translation unit, the compiler can't prove anything about how it's used, which is the case in pretty much all of the examples mentioned in this thread.

You would have to call such a function between the memset and the first time you write to a member. Otherwise the compiler is allowed to say "I put the padding back, and you can't prove otherwise".

> Just because reading the padding of some struct is undefined behaviour doesn't mean accessing those bits by some other means is also undefined.

It's not always undefined, but it says very clearly that the value of padding becomes unspecified.

> they almost always pass a reference to a routine in another translation unit

> you're massively overstating the relevance of compilers eliminating dead stores done via memset

Unless inlining happened, or link-time optimization, or, or...

If the compiler zeroes the memory most of the time, that makes it even scarier. Because all your tests come back clean and safe, then four years later a macro changes and suddenly you're leaking data all over the place.

I don't think I'm overstating the relevance at all. Any security feature that could disappear because of a reasonable, trying-to-help optimization is one that should have a bright red warning label. And this is such a feature. It doesn't require a "sufficiently smart" compiler, and it doesn't require a malicious compiler. This is the kind of thing that can break by accident and ruin everyone's month.

> This is the kind of thing that can break by accident and ruin everyone's month.

Not in practice. Compilers make use of undefined behaviour to optimize things that are widely applicable and profitable. No real compiler does what you're saying and no future compiler is likely to without explicitly being asked to.

I agree that, by the letter of the spec, you're right, but you're still most certainly overstating the relevance.

I'm not arguing that this isn't a real problem or that people shouldn't use memzero_explicit() (or similar) where security is on the line, as I already said several times in another sub-thread -- I'm just saying that this kind of thing is extreme language-lawyering beyond the realms of probability. It's still not an excuse to be lax, but let's be realistic about the actual likelihood of it happening.

It's just a type of dead store elimination. And objects are initialized so often that I could easily see it happening in the future. But I guess we'll just disagree on how likely it is.

> It's just a type of dead store elimination

It's not just any, typical kind of dead store elimination. It also would require other optimization passes that I can assure you no mainstream compiler actually does. You can disagree with me if you want, but you're simply wrong.

Lots of mainstream compilers already have passes that check if every field in an object is definitely assigned. They use this to provide errors or warnings.

That step is 90% of the work. Once you can do that, it's straightforward to assess that every field is assigned after a memset, with no intervening reads, and then remove the memset.

> That step is 90% of the work

I spent several years working on a production grade compiler and I can assure you it's not. But keep just making things up off the top of your head if it makes you feel smart.

Would you care to explain why?

Let's look at a basic common case, as a checklist.

1. All fields are definitely assigned.

2. No functions are called except intrisics.

3. Nothing reads from the object on any control path.

4. Memset happens between creation and first assignment.

We agreed that step 1 is a solved problem, right? Are any of 2-4 difficult? Did I miss any prerequisites for the optimization?

Once you can prove 1-4, isn't the optimization pass as simple as looking for memsets applied to structs, checking 1-4, then deleting the call?

If you're relying upon the value of padding, you're already into undefined behavior.

If you're either using via either assigning or reading the value that is stored in the padded area of a class or struct you are in undefined behavior, any version of the standard you choose. So, care to explain the downvotes?

Although I did not downvote, I suspect your comment is being downvoted because it appears to overlook the risk of data leaking to an adversary through the padding bytes.

If you pass partially uninitialized objects (including padding) between kernel space and user space, there's a chance that the (less privileged) user space code can recover information it shouldn't be able to see, regardless of what the spec says about undefined behaviour. Such information may include secrets that previously occupied the memory where a new, unrelated struct now resides. The same can also apply to passing objects between different machines.

All "undefined behaviour" means is that the spec can no longer guarantee anything about the execution of a program once a constraint is violated. It doesn't say anything about the practicality of actually doing it on any given implementation.

FWIW I didn't downvote you, but your reasoning about undefined behaviour seems overly simplistic. It should never be used to explain away security concerns.

It shouldn't be used to explain away security concerns, but it shouldn't be used to overstate security fixes either.

You can memset to 0, and on your compiler it might be secure, but it's not enough to keep your security guaranteed against future compilers.

There's a reason functions like SecureZeroMemory exist. In a similar situation, you try to prevent leaking secure secrets by zeroing memory before releasing it. But the compiler sees you never use the variable again, and optimizes away the zeroing.

The message to take away is not "be less paranoid". It's "be more paranoid".

Right, but I didn't make a case that memset() is any more secure, did I? The parent comment was talking about undefined behaviour as if it's some kind of universal get-out clause.

Your last paragraph strongly implies that hermitdev was using undefined behavior to "explain away" security concerns.

But the problem is that the security concerns and fixes are all undefined here.

So the initial comment is still very right. If you're relying on the padding to be anything in particular, you're in trouble.

> But the problem is that the security concerns and fixes are all undefined here.

No one in this sub-thread has mentioned (or implied) any "fix". You appear to be putting words in my mouth.

> So the initial comment is still very right.

Of course it's right, but it's also misleading when read in the context of the parent comment.

> If you're relying on the padding to be anything in particular, you're in trouble.

No shit.

> No one in this sub-thread has mentioned (or implied) any "fix". You appear to be putting words in my mouth.

I'm talking about the comments hermitdev was replying to that were treating memset as a 'fix'.

And 'fix' is shorthand for the opposite of "open[ing] you up to all kinds of info leaks". I don't think that's putting words in anyone's mouth.

I guess we are misunderstanding each other's point. I just found hermitdev's comment to be misleading (despite being correct), but perhaps it's just my reading of it.

To be fair though, memset() usually IS a fix. As mentioned by the kernel memzero_explicit() docs:

> usually using memset is just fine (!)

-- https://www.kernel.org/doc/htmldocs/kernel-api/API-memzero-e...

A conforming C compiler can't just remove memset() as it pleases. The case that most often requires memzero_explicit() is when zeroing an object after destruction, because the compiler thinks it can statically determine that it's a dead store. It very rarely happens that a compiler elides a memset() used for initialization.

I'm not sure why you seem to think that memset() can just be dropped at will for no reason whatsoever or that it's somehow always undefined behaviour.

A conforming C compiler can remove a memset that has no side effects.

You mean it can remove a memset() that doesn't cause the observable behaviour to change?

For the sake of argument, can you show me some example code where it would be conforming to remove a memset() call? Preferably a realistic example and not a Google'd copypasta. Because it's all too easy to just regurgitate things you heard and think you understood, but no so easy to demonstrate it yourself.

I agree that it would be rare for a memset() used for initialization to be removed, but this is a good recent paper that gives lots of examples of when stores are eliminated in practice:

Dead Store Elimination (Still) Considered Harmful

https://cseweb.ucsd.edu/~klevchen/yjoll-usesec17.pdf

That doesn’t check whether that memset call writes 0xFF bytes in the padding.

The compiler could also determine that &a.x)[1] is undefined behavior in C (reading past the end of an object), and take advantage of that.

The way to check that is by disassembling the code, and even that doesn’t say what a C compiler should do, only what these particular compilers do.

I feel like you're missing the point (the gcc -O0 was there for a reason) but I updated it. Is this better?

Sending structures that include padding or that you don’t know the exact layout of over the network is another thing you shouldn’t do, period.

Code doing that may break with a compiler change or a compilation flag change on one end of the connection, even if you can guarantee both ends use the same CPU.

This kind of issue happens all the time with structs that a kernel, driver etc. expose to userspace: https://j00ru.vexillium.org/papers/2018/bochspwn_reloaded.pd...

Not sending the structure over the network is insufficient, and rigorous memset is a clear solution to this problem.

Also, C structure layout for a specific ABI is strictly determined; if that weren’t the case you couldn’t compile two C compilation units with different flags/compilers and expect them to work.

Also don't expect memset() to actually be a library call at all.

And most importantly dont expect memset to be executed at all. Most -O2 compilers happily optimize the memset away if if doesn't care about the sideeffects. Therefore there exists the "secure" memset variants, which are the insecure variants adding a simple compiler barrier, so it isn't optimized away.

And there exists the very few really secure memset_s variants with a real memory barrier which guarantees memset to be executed in load and store order, flushing the cache, so that Spectre attacks are mitigated. security-relevant crypto libraries, like openssl and variants don't care about that, because memory barriers are "too slow". only the Linux kernel does it properly. It's still a mess.

you can write your own memset that bypasses all of that. Its just 2 lines of code.

And that will be slower than calling memset() unless you have a really smart compiler.

I don't trust any memset at all. Literally every single memset I looked at was either broken, too slow or insecure. Mostly the wellknown glibc, freebsd, msvcrt and compiler implementations.

That's quite a bold claim. Care to explain further?

I already explained it. Noone implements memory barriers in its safe variants, only compiler barriers. And the trivial variants work byte-wise, not word-wise, which is usually 8x slower.

> Also, C structure layout for a specific ABI is strictly determined

That's true, but when sending stuff over the network you don't know if the receiver uses the same ABI.

Changing compiler or flags won't change structs, unless you do something crazy. If it did, you couldn't link C libraries compiled with one compiler with programs compiled on another.

You can't be sure you can send such structures to different OSes or CPUs however.

> If it did, you couldn't link C libraries compiled with one compiler with programs compiled on another.

You cannot do this unless the compilers have the same ABI, which is not part of the standard.

It depends upon the architecture and compiler. One C compiler for the MC68000 might have int as 16 bits (since externally, the MC68000 had a 16-bit external bus) but another one have ints as 32 bits (since the MC68000 can handle 32-bits internally). This was more of an issue during the 80s and 90s when you had more commercial C compiler available than today.

That is what stdint is for.

> Sending structures that include padding or that you don’t know the exact layout of over the network is another thing you shouldn’t do, period.

What if you do know the exact layout and it happens to have a gap between fields?

Tbh I'm not a network person, but my naive brain says instead of casting to char* and writing to the buffer, couldn't you just write out individual fields to a buffer so you control every byte? That seems smarter and safer.

It depends. It was common case for years in games to just cast your networked data to an array and send over the network. Nowadays it's much more in fashion to pick a serialization library to sit in between, as PCs are faster. But one might choose not to depending on the problem space. Embedded devices might not have the spare cycles or the space for a full serialization library.

So...you're not wrong, but it's basically the same reason people use C over something with more ergonomics.

Edit: the last time I wrote a networking library, I also just cast to raw structures, mainly because it was less work (didn't want to integrate a serialization library on both sides of the link). This was fine for a while (and very fast), but also hid some bugs in the protocol, and made sending arrays of data very ugly. Eventually we put protobuf on top, and things ended up much nicer.

How does sending raw memory to the socket work if you have a struct filled with pointers?

I think they simply don't do that with structs that contain pointers?

Indeed, this. If you have data with pointers, you either need to flatten (read: copy) your structure into something appropriate for structuring, or just get a proper serialization layer.

I guess there could be some fun workarounds with ptrdiff_t instead of pointers. "My child is 1000 bytes that way in memory" etc.

That's a pretty classical file serialization trick to pack a bunch of heterogeneous data in one file. A header/manifest starting the file, with an array of names, offsets, and sizes.

Actually works somewhat well for mmap'ed access, too, by using a macro or function to "dereference" "pointers" (offsets).

Or even an OffsetPtr class with overloaded arrow and dereference operator, if you swing that way. :)

What’s the runtime performance of that in-practice? Does x86, x64 and ARM have support for relative offset pointers without any performance impact? And do compilers use them?

Yes, address generation can at the very least do [reg + constant], and on x86, much much more. These are bread and butter optimizations constantly used by compilers all over the place, because this is how field access works.

Do you write a separate serialization function for every single struct and sub-struct you send over a network? And then continue maintaining every single one to keep it consistent whenever a field is added/removed/modified in any struct?

(Though I should mention that macros actually do make this doable, but they're hardly pretty, and I have yet to see many people embrace this approach, if they realize it's possible at all.)

> Do you write a separate serialization function for every single struct and sub-struct you send over a network? And then continue maintaining every single one to keep it consistent whenever a field is added/removed/modified in any struct?

Yes? Of course you do that? Or pick one of a hundred libraries that do that for you - protobuf, cap'n proto, flatbuffers, etc...

That's the protobuf approach, but things like Capnp are different, right? Isn't its wire format its in-memory format? I thought it was closer to the "straight memcpy" approach.

Protobuf serialization functions can turn into a lot of generated code too (read: big binaries) and stomp all over your icache, because they're different for each type. More "data-driven" approaches using type-generic functions with some runtime type-specific data (like proto descriptors&reflection) can be a good tradeoff for code that isn't as "hot". Typically more branchy than a memcpy or a type-specific function though.

There are dozens, if not hundreds of serialization libraries. None of them are going to compare to something like sending a raw struct on the wire, if you need absolute performance. Probably the biggest advantage of things like grpc, flatbuffers, capnproto, etc, are they provide serialization libraries for many different languages. Sending a raw struct will be up to the implementer to get the other side correct.

And by sticking with the serialization libraries with higher performance, like flatbuffers and capnproto, you also lose a little bit of future flexibility, since its typically harder to change the format of these structures compared to higher level libraries.

Author of Cap'n Proto here...

> None of them are going to compare to something like sending a raw struct on the wire, if you need absolute performance.

I don't know about that.

For simple flat structs containing integers only (no pointers), Cap'n Proto is nearly identical to raw structs.

For complex structures with pointers, sending raw structs doesn't work.

Admittedly you may suffer unwanted code bloat linking in the Cap'n Proto library if all you really want to do is send a flat, raw struct.

> And by sticking with the serialization libraries with higher performance, like flatbuffers and capnproto, you also lose a little bit of future flexibility, since its typically harder to change the format of these structures compared to higher level libraries.

I disagree with this. Cap'n Proto's compatibility story is essentially the same as Protobuf and JSON: You can add new fields and old programs will ignore them.

(I suspect Flatbuffers is also similar but I haven't looked at it in a long time.)

That's what I do these days for simple stuff, or for stuff which needs to talk to nonstandard hardware (eg. talking to a PLC, where I need to know the exact bit layout of the packet in order to unpack it manually on the other side).

If you're doing a large project on supported platforms it's probably worth using a pre-existing serialisation library like protobuffers or something.

There is an attribute that lets you specify alignment per-struct, which means you will know the exact layout.

But it is still potentially problematic if not all of your clients have the same endianess. Memcpy and exact layout still fails you there.

RDMA (Infiniband) does DMA to/from network. Why we should not use RDMA?

Hmm, that concern is not just limited to RDMA; any kind of shared memory communication across a trust boundary is vulnerable to info leaks in the padding of the struct. That does seem like a real issue.

Not sure what's the appropriate mitigation. Having to memset is a drag, and it's easy to forget to do it in some place or another. I'm tempted to say that you should only use packed structs for shared memory communication across trust boundaries.

Well, Infiniband is intended mainly for supercomputers. So security is of no real concern within the Imfiniband network as that is implicitly a trusted environment. Access comtrol to that network is the main security boundary.

Edit: Also, lots of simulation codes I am aware of tend to send flat arrays anyway. So I'd say that your concerns are mostly theoretical in nature.

RDMA, both roce and infiniband have some significant limitations, specifically at higher rates. But they're the best we have right now for direct dma.

Serialization is a non-trivial problem. Especially when dealing with different OS, network stacks, and architectures. protobuf alleviates this quite a bit.

No, but the follow up article is called Stop Memcpying Structures. And the follow up to that Stop Memcmping Structures. :)

You're probably not serious, but why waste time and complexity copying structure elements into a buffer (which likely needs to be memset/bzero'd!), if you can just directly use the struct, all consistent with the standard?

C is not a very high-level language. Apart from where undefined behavior is called out, you can and should assume how data is represented on a low level. It was made for writing operating systems and their kernels after all, where interfacing in a bit-accurate way is common.

I'm joking because endless flame wars have already been spent debating this issue. But I realize not everyone has heard them yet, so let's strap on the football shoes and bring out the horse carcass one more time!

There are platforms on which NULL, integer 0 and floating point 0 is not the same. Hence memsetting won't do the right thing. On modern platforms they are, but still, you don't want to get the Standard C Weenies on your back - they're even more annoying than the Rust Evangelism Strike Force. You don't want demons coming out of your nose, do you? :)

They do have a point, though. Suppose you are handling a struct you thought were memsetted but was not (your stupid team mate used those damn initializers!), then when serializing it you will get undefined bytes in the stream, possibly triggering hash values to mismatch causing mysterious rebuilds and other stuff.

Wrt to field accesses, yes those can overwrite padding bytes. Suppose a is an 8 bit char field, but padded to 4 bytes. Then "foo->a = 22" could be translated to "MOV [RSP+16], EAX" which overwrites the 3 padding bytes with undefined values. The solution, as you mention, is to use packed structs but in those field accesses are much less efficient.

Bottom line: code whichever way you want. :) Me, I like to live in the fast lane from time to time and those memset/cpy/cmp functions are sooo handy. :)

By the standard, the null pointer must be equivalent to 0.

Only if you use conversion within the language: void *p=0;

If you said memset(&p, 0, sizeof(p)) then that's not guaranteed to work on all implementations.

gizmo686's comment [1] explains the subtle difference:

> [...] The technicality of the memset example is that it does not set the bits of the pointer by referring to it as a pointer, so the requirement that 0 behave as if it was a null pointer does not apply.

[1] https://news.ycombinator.com/item?id=19767316

The memory representation of the null pointer is implementation-defined. The C99 standard only guarantees that a integer constant expression with the value 0, implicitly or explicitly cast to a pointer type, will give a null pointer. It does not guarantee that a null pointer cast to an integer type (other than _Bool) will give the value 0; all pointer-to-integer casts are implementation-defined. It also does not guarantee that a non-constant expression which evaluates to 0 will produce a null pointer if cast to a pointer type.

In short, there may exist counter-intuitive but nonetheless standards-compliant platforms where tests such as "assert((uintptr_t)(void * )0 == 0);" or "int x = 0; assert((void * )x == (void * )0);" would fail.

Did you mean this as a joke? I can't find a follow-up article (this one was written today).

Please explain how you send a struct over the network without UB and leaking the padding.

Use a packed struct (e.g. __attribute__((packed))). You may take a performance hit due to lack of alignment, but that's the judgement call

__attribute__((packed)) is not part of the standard, and therefore doesn't quite answer the question. I believe the standards-compliant way of doing this is that "you can't".

Would this work

  struct S {char data, char pad0, char pad1, char pad2};
  int main() {S dataStruct; printf("%s", dataStruct.data); return 0;}

and give me a 4-byte struct or would the compiler optimize it all away and leave me with a 1-byte struct?

Since you don't actually access the size of the structure, I see no reason why it matters. Also, FWIW, your code has undefined behavior because you call printf with the wrong type.

Yes on the UB, I'm passing a "char" into a "char *", sorry about that. I don't pass lone char data that often.

Well, I chose this example specifically to ask whether the resulting memory structure, completely independent of calling a sizeof(), would still be as if I were to call a sizeof() (which then I believe would be 4 bytes large) if not accessing the total size of the struct or any of the padding members.

It makes a difference in the memory footprint, which can become important, if you're programming a device with just 64-bytes total RAM.

> performance hit due to lack of alignment

This isn't true of modern processors.

Sure you get free unaligned access for scalars on x86, but unaligned arrays are still trouble if you use SSE (which basically everyone does).

By knowing its exact size with sizeof(), and using that and a cast to uint8_t* to both bzero() it out before use and memcpy()ing it when sending it. Unless the C standard says that the compiler may change the padding bytes of the existing storage?

To be clear, it is something like this, which is not bad at all to write:

  struct st s;
  bzero(&s, sizeof(s));
  // ...
  write(fd, &s, sizeof(s));

I’ve dealt with many systems that have tons of unnecessary serialization logic, and found them slower and buggier than just letting the C spec define the data format.

Having said that, it is possible to get this sort of thing right with C++ and templates (thanks to inlining).

Higher-level languages without compile time code generation (like Java and Python) usually make serialization a train wreck in comparison with “send raw structs” or “make the [C++] compiler generate and typecheck the machine code that does the serialization”

There's enough wiggle room in the standard that two compilers would be happy making two compliant but still different physical layouts for the same struct declaration. I'm not even worrying about cross-platform issues or bitfields yet.

You can get away with directly writing a struct for simple structs, but eventually it will bite you.

Packed structs are reasonably standard (even gcc and visual studio can be made to pack in the same way using the same syntax these days).

The remaining issue is endianness, but you can configure ARM to match x86 at boot so shrug.

Also, if I tell any reasonably experienced developer my protocol sends a packed struct with these fields:

  uint8_t version
  uint16_t reserved
  uint8_t op
  int32_t status
  uint32_t data_length
  // void data[]

They’ll be able to write a correct deserializer in pretty much any language, and can probably write it in ~ 10 lines of code.

I agree that packed structs can be tricky, but they’re much better than the vast majority of serialization libraries I’ve dealt with (including all of the open source ones, and I’ve used all the usual suspects).

> The remaining issue is endianness, but you can configure ARM to match x86 at boot so shrug.

ARM in BE mode is pretty rare / nonexistent. Small MIPS chips still float around in mostly low-end networking gear, though, and those do tend to run in BE.

My experience with packed structs is there really aren't any gotcha's. Either works or not.

The big annoyance is network people all use big endian probably just to spite everyone else.

Why would it be to spite everyone? Big endian is network order by definition. It's hard-coded into every asic in switches and routers for decades. If you don't use big endian your packets won't even make it to where you want. Of course, your own payload can do whatever you want as long as you know with 100% certainty nobody else will ever want to talk to your application and get confused.

Consider the lorawan spec which was written in the last 5 years. The stack is designed to be implemented in software on small 32 bit microcontrollers which are _all_ little endian. And it's big endians.

The people that wrote that spec knew that and went with big endian anyways.

Also I work with people that do network hardware. They can't care less about little or big endian. Makes no difference to them.

Someone needs to pick one or the other: you can't have it both ways.

Yeah they are picking the side everyone else abandoned 25 years ago.

In the "..." part you presumably meant to include writes to members of s, and those writes are allowed to change padding bytes and leak information.

A small quibble: bzero() is not defined by the C standard. memset() is.

Obligatory pet peeve note: sizeof is a unary operator whose argument is written in parentheses if it is the name of a type. It is not a function.

All answers to this must be wrong as it's simply not possible to do it. You need to take endianness of each field into account, and you cannot do that by treating the struct as a single blob.

But if you're slamming in-memory structs over the wire, you're using packed structs. You explicitly don't want padding when you use structs that way.

Whether you should do this in 2019 is another question, but sometimes it's still the easiest way to implement a binary protocol.

And you're not writing code for big-endian machines to interoperate with little-endian machines.

It's easy to write binary protocol code that way but it isn't portable.

It goes without saying that you have get the endian right when you store the values. That's what the endian macros are for.

Reply on news.ycombinator.com