Hacker News Reader

Comment by xg15 | Hacker News Reader

> And when I say truly, I mean for platforms without file systems

Are we're really talking about compiling on such platforms? And if that's the case, how would #include work but not #embed?

View on HN

No, I'm mainly talking about targeting. My point is not so much about embed, but rather that, almost anything you assume you think you know about how computers work isn't necessarily true, because C targets such a wide group of platforms. Almost always when some one raises a question along the line of "No platform has ever done that right?", some one knows of a platform that has done that, and it turns out has very good reasons for doing that.

For this reason, everything is much more complicated then you first think. For me joining the WG14 has been an amazing opportunity to learn the depths of the language. C is not big but it is incredibly deep. The answer to "Why does C not just do X?" is almost always far more complicated and thought through than the one thinks.

Everyone in the wg14 who has been around for a while, knows this, and therefore assumes that even the simplest addition will cause problems, even if they cant come up with a reason why.

Yeah, but then I have to side with the author - how could a compile time only feature which doesn't even introduce new language semantics possibly be affected by the multitude of build targets?

Unless "it's more complicated than you think" is the catchall answer to any and all proposals for new language features. In which case, how to make progress at all?

Also, I find the point about the language being "truly portable" a bit ironic, considering the whole rationale of #embed was that the use case of "embed large chunks of binary data in the executable" was completely non-portable and required adding significant complexity to the build scripts if you were targeting multiple platforms.

It's easy to make a language portable on paper if you simply declare the non-portable parts to not be your responsibility.

> Everyone in the wg14 who has been around for a while, knows this, and therefore assumes that even the simplest addition will cause problems, even if they cant come up with a reason why.

That's not something to be proud of.

> That's not something to be proud of.

Its learning from old mistakes.

Look at embed as an example. Look how complex it is, dealing with empty files, different ways of opening files, files without lengths, null termination... the list goes on. This is typical of a proposal for C, it starts out simple "why cant i just embed a file in to my code?" and then it gets complicated because the world is complicated.

I worry a lot about people loading in text files and forgetting to add null termination to embeds. I would not be surprised if in a few years that provides a big headline on Hacker news, about how that shot someone in the foot and how C isn't to be trusted. The details matter.

> I worry a lot about people loading in text files and forgetting to add null termination to embeds. I would not be surprised if in a few years that provides a big headline on Hacker news, about how that shot someone in the foot and how C isn't to be trusted. The details matter.

The compiler should insert the null terminator if it's not in the embedded file.

I don't think adding a null terminator is useful for binary files which are not null-terminated strings, and may even have embedded 0 bytes in the middle.

sure, but if it's a string that requires it to be null terminated, there's no reason the compiler can't solve that problem

How does the compiler know if an array is a string or not?

It doesn't have to, you just add a zero byte at the end of the embedded byte sequence in the object file. It's up to the programmer to make the choice how to interpret that.

That said for binary embeds you almost always need the length embedded as well, which has been the case for every tool I've used to embed files in object code. You usually get something like

    const size_t My_FILE_LENGTH = ... ;
    const uint8_t MY_FILE[] = { ... };

> It doesn't have to, you just add a zero byte at the end of the embedded byte sequence in the object file. It's up to the programmer to make the choice how to interpret that.

Like, are you proposing that the ideal semantics for `#embed "foo"` if foo contained 0x10 0x20 0x30 0x40 to be to expand to `16, 32, 48, 64, 0`? That seems more annoying than the opposite, given that:

> That said for binary embeds you almost always need the length embedded as well, which has been the case for every tool I've used to embed files in object code. You usually get something like

TFA demonstrated it by relying on sizeof() of arrays doing the right thing:

    static_assert((sizeof(sound_signature) / sizeof(*sound_signature)) >= 4,
        "There should be at least 4 elements in this array.");

You'd need to change this to subtract one every time, which sounds more annoying when embedding binary resources than adding the zero to strings would be.

This is another issue here. If loads of compilers start doing this then programs start relying on it an then it becomes a de-facto undocumented feature. That means if you move compilers/platforms you get new issues. A lot of what the C standard does is mopping up these kinds of issues.

Then require compilers implement it in the standard. I think it's really backwards to ignore the tool chain and its ability to prevent bugs from entering software.

It's stuff like this that leaves us writing C to rely on implementation defined behavior. Under specification that leaves easy holes to fill will be filled by the compiler and we will rely on them. Just like type punning.

This is the problem. Things get complicated fast. If we mandate null termination, then its impossible to have multiple embeds in a row to concatenate files, or we need some how to have rules for when to add null termination and not. These rules in turn are not going to be read by all users, so some people will just assume that embed always adds null terminate in when it doesn't and then we are back to square one. The more we add the more corner cases there are.

Why assume the data should be null terminated? Its an array with a known compile time size. Binary data often needs to include 0 / NULL.

This is literally the premise of this comment chain. Can you see how the standard can get dragged around by various competing needs? People want to embed strings, with null termination, some just want to embed binary data. Neither is particularly “wrong”. And in this thread everyone is presenting “obvious” solutions that are one or the other.

No, at least in this case I think the feature chooses the correct path on this. If you want to embed a string that’s null terminated then just save the string into a file as a null terminated string or copy it from the array to a proper string type. It looks like you may even be able to use this with a struct which might let you add a null termination after the array.

Though yes I agree lots of features get bogged down trying to handle everything. But in this case not adding it creates even more complexity. You can store strings natively in C. You can’t include binary blobs in C in a platform independent way so you have hacks that explode compile times for everyone using that software.

The ability to add vendor specific attributes also allows for those use cases to evolve naturally while still solving the core problem of embedding binary data.

I, FWIW, agree with the implementation. I'm just saying that this thread has someone saying "the compiler should add null termination" in it so the choice to make is not obvious.

I don't see how that would prevent sequential embeds unless you have defined the semantics of embed to be a textual include, which if that's the case then the mistake are the semantics and not the constraints on compilers or embedded files.

edit: why is it desirable to concatenate files with #embed? Does that not seem out of scope if not contrived?

If you need 0 termination simply write

    const char foo[] = {
    #embed <file.txt>
        , 0
    };

What ? No it shouldn't. Use pointer+size if you want strings

I worry a lot about people loading in text files and forgetting to add null termination to embeds.

If you really worry about that, why did you vote in favour of this feature (as you stated earlier)?

He’s just trying to justify why it took 5 years to approve a feature that is so similar to #include.

null termination is not to be added to embed. embed adds a const sized buffer of unsigned bytes, not strings. files are not strings, files do contain \0.

and I still don't get why embed is so much better than xxd included buffers. it's more convenient sure, but 10x faster?

#embed is faster because it skips the parsing step. xxd generates source code tokens, which the parser must then decode back into bytes; embed just reads the bytes directly.

Multiple big-O steps faster.

That's not how big-O works.

I was on X3J11, the ANSI committee that created the original C standard and my experience was similar. It was a great opportunity to learn C at depth and get an understanding of many of the subtle details. We rejected a great many suggestions because our mandate was to standardize existing practice, address some problem areas, and not get too creative. (We occasionally did get too creative. The less said about noalias the better.)

We are still fixing bugs in restrict...

Maybe you can answer a question I have: what companies are still supporting C compilers for sign-magnitude and 1s-complement machines today? I've been programming for almost 40 years now, and I have never come across any machine that is sign-magnitude or 1s-complement (I have encountered real analog computers---a decent sized one too---about 9' (3m) long, 6' (2m) high, and about 3' (1m) deep, requiring hundreds of patch cables to program).

Reply on news.ycombinator.com