Embed is in C23
thephd.dev
thephd.dev
Cs main strengthen is its portability and simplicity. Therefore we should be very conservative, and not add anything quickly. There are plenty of languages to choose form if you want a "modern" language with lots of conveniences. If you want a truly portable language there is really only C. And when I say truly, I mean for platforms without file systems, or operating systems or where bytes aren't 8 bits, that doesn't use ASCI or Unicode, where NULL isn't on address 0 and so on.
We are the stewards of this, and the work we put in, while large, is tiny compared to the impact we have. Any change we makes, needs to be addressed by every compiler maintainer. There are millions of lines of code that depend on every part of the standard. A 1% performance loss is millions of tons of CO2 released, and billions in added hardware and energy costs.
In this privileged position, we have to be very mindful of the concerns of our users, and take the time too look at every corner case in detail before adding any new features. If we add something, then people will depend on its behavior, no matter how bad, and we therefor will have great difficulty in fixing it in the future without breaking our users work, so we have to get it right the first time.
Thank you especially for reminding everybody that programming is much more than web programming and information systems.
Its also worth remembering that a lot of higher level languages have runtimes / VMs are implemented in C. Web applications rely heavily on databases, java script VM, network-stacks, system calls and operating system features, all of which are impemented in C.
If you are a software developer and want to do something about climate change, consider becomming a compiler engineer. If you manage to get a couple of tenths of a percent performance increase in one of the big compilers during your career, you will have materially impacted global warming. Compiler engineers are the unsung heroes of software engineering.
Anything that includes its own memory allocator (that doesn’t call malloc()) is probably not implemented in standardized C.
Ask a question on Stackoverflow about code that requires nonstandard flags or uses UB. You will be told your program is not really in C/C++ and that your question makes no sense. You will be lectured on nasal demons for the nth time.
File a bug against a compiler asking the optimizers to back off using UB to subvert the intentions of the programmer and you will be told there's no way to even know the intentions of the programmer given that the standards don't apply.
So we can all move to a nomenclature where C is used loosely to indicate a family of languages, once of which happens to be standardized. I'd be happy to do that.
But let's not bait and switch, using standards pedantry to dismiss feature requests and bug reports, but then turning around and saying C/C++ is suitable for implementing runtimes of other languages when nothing can really be achieved in that space without going beyond the language spec.
And really in 2022 runtimes, JITs, GCs should be the primary use of C/C++. Many other uses (systems software, compilers, desktop apps, phone apps) are not suited for C/C++ due to security, stability, ease of development and newer languages that make more sense for these domains.
I've heard this kind of claim a number of times and I think it's more complicated than the crude statistical measurement makes it sound. Personally, I think that most programs are not run frequently enough to matter from an emissions perspective. For programs that are, like ML training programs, users will just train more data if the algorithms are faster so most energy efficiencies will get wiped out by the increased usage.
Even if that theory is wrong, what if there is a language that is 10% better than C for 95% of common C use cases? Wouldn't it be better for compiler engineers to focus on developing that language than micro-optimizing C?
Source: Rationale for International Standard — Programming Languages — C https://www.open-std.org/jtc1/sc22/wg14/www/C99RationaleV5.1...
I don't know if this rationale is still followed, but I think it applies here. We need to be cautious when adding new features to C.
you cannot find non-normalized substrings (strings are Unicode nowadays), utf-8 is unsupported. coreutils and almost all tools don't have proper string (=Unicode) support.
This seems totally misconceived to me as a basis for standardizing a language in 2022. You are optimizing for the few at the expense of the many.
I get that these strange architectures need a language. Why does it have to be C or C++? They can use a nonstandardized variant of C, but why hobble the language that is 99% used on normal hardware with misfeatures that are justified by trule obscure platforms.
I would like to caution you against thinking that these weird platforms are old machines from the 60s that only run in museums. For instance many DSPs have 32bit bytes (smallest memory unit that can be individually addressed), so if you have a pair of new fancy noise canceling headphones, then its not unlikely you are wearing a platform like that on your head everyday.
I would love a new language for implementing high level languages. I've worked on several of these projects and we use mostly unstandardized dialects of C++ and it's really not fit for purpose.
I'm thrilled that the folks steering C are keeping the needs of embedded developers at the forefront.
What does this even mean? Zig very much has embedded use as a target as well, with freestanding use being a first-class citizen. The majority of the standard library works without an OS (and if you wish, you can provide your own OS-like interfaces quite easily, and use the OS-dependent parts in a freestanding environment). I've written a UEFI bootloader in Zig, and right now I'm using it on an RP2040 as well, cross compiling from an M1 Mac without the need to install any additional cross compilers.
I'd argue that it might be even better for embedded than C/C++ eventually, as unlike them, allocation is even more tightly controlled, with a convention that only functions (or datastructures) that take an allocator as an argument will allocate memory on the heap. Future versions may even restrict recursion in certain contexts to guarantee max stack depths.
They chose to build Zig on top of the worst compiler backend toolchain for embedded support.
Does GCC target any of those in its standard build? Oh. Wait. It doesn’t. It builds for only one architecture at a time and if you want anything else you’ll need to recompile the compiler.
LLVM is even intended to be optional in future, and the Zig team is well on their way to making that a reality with the work on stage2.
What backend would you suggest for embedded support?
You _cannot_ simply recompile LLVM to support the targets I listed without using unsupported, risky patches from community sources; if any such patches exist. The multiple architecture support in one binary is a pointless feature when installing multiple GCC builds only takes a tiny, insignificant fraction of my development machine's disk space. Never in my life have I thought "Wow, I'd really speed up development if GCC was one enormous binary holding all possible targets."
Zig's best path forward is to support targeting C as a first tier target; which they seem to be interested in doing.
The potential change would make these types address-sized, so you can't fit a whole pointer in them on Morello where the pointer isn't just an address. On other platforms this change makes no practical difference.
Like, it doesn't even cover ESP32 chips, despite attempts to get support into LLVM since 2019.
point being, they can't leverage LLVM
Isn’t C99 an option? Why can’t more advanced things go into newer C and people who genuinely need something more basic can use C99.
The reality however is that you cant escape never versions entirely. Not all code you interact with was written in the subset you want, so when your favorite OS or library starts using header files with newer features you need to run that version of the language too.
Another less appreciated detail, is that a lot of WG14 work is not about adding new features but clarifying how existing features are meant to work. When the text is clarified this gets back-ported to all previous versions of C in major compilers. An example of this is "provenance". This is a concept that implicitly been standard since the first ISO standard, but only now is becoming formalized. This means that if you want to adhere to the C89 standard, you will find a lot of clarifications about how things should work in the C23 standard.
> so if you have a pair of new fancy noise canceling headphones, then its not unlikely you are wearing a platform like that on your head everyday.
Chip shortage aside, the likelihood of these devices using obscure hardware like discrete DSPs is going down as cheaper low power architectures are becoming commoditized.
My point is that a lot of this sentiment is treating C as the portable backend of a compiler because there is no portable front end. It holds us back in a lot of ways from iterating on systems languages in ways that are interesting and valuable.
Ideally there would be an IR with stable textual and binary format that can be compiled into the machine code for various ISAs and support the extensions necessary by silicon manufacturers for the exotic bits. I've used exotic tool chains for weird ISAs where the exotic bits are sugar added to a GCC front end, and it always feels wrong and limiting.
Sure, but it's the same line of reasoning that made C relevant in the first place, and keeps it relevant today - some library your dad wrote for a PDP-whatever is still usable today on your laptop running Windows 10.
Because it's antiquated, it's also extremely easy to support, and to port to new and/or exotic platforms.
No, it isn't. Go on. Go ahead and try
See it break in a million weird ways. (Or, for a start, it will have the K&R C format, which is a pain to maintain)
"If your computer doesn't have 8-bit bytes" at this day and age? It belongs in a dumpster, sorry.
(I think the only "modern" arch that does this is PIC, and even only for program data - where you're not running anything "officially" C89 or later)
Compiling with -Wall -Werror is pretty much standard those days.
For code to be miscompiled, there has to be a definition of what correctly compiling it would mean, and if there were, it would not be undefined behavior.
Eg the fact that overflowing a signed int can cause the compiler to go amuck would certainly be a surprise to the person who wrote code for the PDP-11.
My point was that this niche doesn't cover the entirety of the embedded / IoT space.
I was solely replying to the commenter who said that all reasonable modern systems have filesystems so I put one in for the embedded software developers.
I don’t even like programming in C but I respect what the committee is trying to do, and yes I do sometimes write C code.
C itself carries a lot of assumptions about computer architecture from the PDP-9 / PDP-11 era, and this does hold current hardware back a bit: see how well the cool nonstandard and fast Cell CPU fared.
A language standard should assume as little about the hardware as possible, while also, ideally, allowing to describe properties of the hardware somehow. C tries hard, but the problem is not easy at all.
OTOH you can write stuff like `*src++ = *dst++`, and it would neatly compile into something like `movb (R1)+, (R2)+`, a single opcode on a PDP-11.
Modern toolchains must exist for marginal systems. It's understandable to want to write code for a machine from 1975, or a bespoke MCU, on a modern Thinkpad. It is not necessary to support a modern compiler running on the machine from 1975 / bespoke MCU. You might as well argue against readable diagnostic messages because some system out there might not be able to print them!
The 1970s mainframe this program will run on has no idea that Unicode exists. Fine. But, the compiler I'm using, which must have been written in the future after this was standardised, definitely does know that Unicode exists. So let's just agree that the program's source code is always UTF-8 and have done with it.
Jason Turner has a talk where the big reveal is, the reason the slides were all retro-looking was that they were rendered in real time on a Commodore 64. The program to do that was written in modern C++ and obviously can't be compiled on a Commodore 64 but it doesn't need to be, the C64 just needs to run the program.
For auto I think the argument is that if you poke around in real software the storage specifier was basically never used because it's redundant. That's the rationale WG21 had to abolish its earlier meaning in C++ before adding type deducing auto.
As I read it, N2368 (which I think is what they took?) gives C something more similar to the type inference found in many languages today (which gives you a diagnostic if it can't infer a unique type from available information) whereas C++ got deduction which will choose a type when ambiguous, increasing the chance that a maintenance programmer misunderstands the type of the auto variable.
However it got inference from return, which I think is a misfeature (although I think I can see why they took it, to make generics nicer). With inference from return, to figure out what foo(bar)'s type is, I need to read the implementation of foo because I have to find out what the return statements look like. It's more common today to decide we should know from the function's signature.
This is somewhat mitigated by the fact that N2368 says auto won't work in extern context, so we can't just blithely say "This object file totally has a function which returns something and you should figure out what type that is" because that's clearly nonsense. You will have the source code with the return statements in it.
Are we're really talking about compiling on such platforms? And if that's the case, how would #include work but not #embed?
For this reason, everything is much more complicated then you first think. For me joining the WG14 has been an amazing opportunity to learn the depths of the language. C is not big but it is incredibly deep. The answer to "Why does C not just do X?" is almost always far more complicated and thought through than the one thinks.
Everyone in the wg14 who has been around for a while, knows this, and therefore assumes that even the simplest addition will cause problems, even if they cant come up with a reason why.
Unless "it's more complicated than you think" is the catchall answer to any and all proposals for new language features. In which case, how to make progress at all?
Also, I find the point about the language being "truly portable" a bit ironic, considering the whole rationale of #embed was that the use case of "embed large chunks of binary data in the executable" was completely non-portable and required adding significant complexity to the build scripts if you were targeting multiple platforms.
It's easy to make a language portable on paper if you simply declare the non-portable parts to not be your responsibility.
> Everyone in the wg14 who has been around for a while, knows this, and therefore assumes that even the simplest addition will cause problems, even if they cant come up with a reason why.
That's not something to be proud of.
Its learning from old mistakes.
Look at embed as an example. Look how complex it is, dealing with empty files, different ways of opening files, files without lengths, null termination... the list goes on. This is typical of a proposal for C, it starts out simple "why cant i just embed a file in to my code?" and then it gets complicated because the world is complicated.
I worry a lot about people loading in text files and forgetting to add null termination to embeds. I would not be surprised if in a few years that provides a big headline on Hacker news, about how that shot someone in the foot and how C isn't to be trusted. The details matter.
The compiler should insert the null terminator if it's not in the embedded file.
That said for binary embeds you almost always need the length embedded as well, which has been the case for every tool I've used to embed files in object code. You usually get something like
const size_t My_FILE_LENGTH = ... ;
const uint8_t MY_FILE[] = { ... };Like, are you proposing that the ideal semantics for `#embed "foo"` if foo contained 0x10 0x20 0x30 0x40 to be to expand to `16, 32, 48, 64, 0`? That seems more annoying than the opposite, given that:
> That said for binary embeds you almost always need the length embedded as well, which has been the case for every tool I've used to embed files in object code. You usually get something like
TFA demonstrated it by relying on sizeof() of arrays doing the right thing:
static_assert((sizeof(sound_signature) / sizeof(*sound_signature)) >= 4,
"There should be at least 4 elements in this array.");
It's stuff like this that leaves us writing C to rely on implementation defined behavior. Under specification that leaves easy holes to fill will be filled by the compiler and we will rely on them. Just like type punning.
Though yes I agree lots of features get bogged down trying to handle everything. But in this case not adding it creates even more complexity. You can store strings natively in C. You can’t include binary blobs in C in a platform independent way so you have hacks that explode compile times for everyone using that software.
The ability to add vendor specific attributes also allows for those use cases to evolve naturally while still solving the core problem of embedding binary data.
edit: why is it desirable to concatenate files with #embed? Does that not seem out of scope if not contrived?
const char foo[] = {
#embed <file.txt>
, 0
};If you really worry about that, why did you vote in favour of this feature (as you stated earlier)?
and I still don't get why embed is so much better than xxd included buffers. it's more convenient sure, but 10x faster?
Isn't there literally a single GPU for which it is true?
Asking because everytime this surfaces, someone inevitably asks for an example, and the only example I've seen over the years was of one specific (Nvidia?) GPU that uses NULL of 0xFFFFFFFA (or something similar).
That is, do you know how common it is for NULL to not be 0?
It’s not really a problem in practice unless you want to dump the whole flash and some != NULL check somewhere
*(u16)0 = segment // dereference of null!
*(u16)2 = offset
IIRC, the M68k (which was a popular C target with official Linux support) did the same thing.
For a more recent example, AVR (popularized by Arduino) maps it’s registers into RAM starting at address 0. So, if you wanted to write to r0, you could write to NULL instead. Although one would be using assembly for this, not C.
Reading further, I don't think this was ever addressed when someone else brought it up. I cannot for the life of me imagine a system where #include works but #embed doesn't. Again, it's fine if some systems have non-standard subsets of the C standard....why hobble the actual standard for code which can be compiled on systems where you have a filesystem (that will handle #include by the way) for the systems without filesystems?
I don't think it would, you'd cross-compile for it on a platform with a file system. I think the parent poster's point was that C is the only option for some ultra low resources platforms and that a conservative approach should be taken to add new features in general. I don't think they were saying that specifically that not having a filesystem is problematic for this particular inclusion.
Genuine question: why do we want these platforms to live, rather than to be forced to die? They sound awful.
I understand retrocomputing, legacy mainframes, etc; but 99% of that work is done in non-portable assembler and/or some flavor of BASIC; not in C.
I think a lot about that! I'm a member of the UB study group and the lead author of a Technical Report we hope to release on UB.
In short, "Undefined behavior" is poorly named. It should have been called "Things compilers can assume the program wont do". With what we call "assumed absence of UB" compilers can and do do a lot of clever things.
Until we get the official TR out, you may find I made a video on the subject interesting:
If I may gripe about C for a bit though. I do truly appreciate C's portability. It's possible to target a very diverse set of architectures and operating systems with a single source. Still, I do wish it would actually embrace each architecture, rather than try to mediate between them. A lot of my gripes with C are due to undefined behaviour which is left as such because of platform differences. I've never seen my program become faster if I remove `-fwrapv -fno-strict-aliasing`, but it has resulted in bugs due to compiler optimisations. I really wish by default "undefined behaviour" would become "platform-specific behaviour", with an officially blessed way to tell the compiler it can perform farther optimisations based on data guarantees.
C occupies a very pleasant niche where it lets you write software for the actual hardware, rather than for a VM, while still being high level enough to allow for expressiveness in algorithms and program organisation. I just wish by default every syntactically valid program would also be a well-defined program, because the alternative we have now makes it really hard to reason about and prove program correctness (i.e. that it does what you think it does).
https://groups.google.com/a/isocpp.org/g/std-proposals/c/b6n...
Enjoy the naysayers if you like! I'm glad someone spent the time and effort to push past them. Bit too late for me - I have moved on to Rust which had support for this from version 1.0.0.
> There's also the standard *nix/BSD utility "xxd".
> Seems like the niche is filled. Or, at least, if you want to claim that
> (A) XPM
> (B) incbin
> (C) "xxd -i"
> (D) various ad-hoc scripts given in http://stackoverflow.com/questions/8707183/script-tool-to-co...
>...do NOT completely fill this evolutionary niche
> This ultimately would encourage a weird sort of resource management philosophy that I think might be damaging in the long run.
> Speaking from experience, it is a tremendously bad idea to bake any resource into a binary.
> I'll point out that this is a non-issue for Qt applications that can simply use Qt's resources for this sort of business.
(Though credit to Matthew Woehlke, he did point out a solution which is basically identical to #embed)
> I find this useless specially in embedded environments since there should be some processing of the binary data anyway, either before building the application
In fairness there was a decent amount of support. But given the insane amount of negativity around an obviously useful feature I gave up.
I wonder if there was a similar response to the proposal to include `string::starts_with()`...
What a pompous douche whoever wrote that was.
> > This ultimately would encourage a weird sort of resource management philosophy that I think might be damaging in the long run.
So, this might be a valid point, although not enough to reject the feature for. It true that it's a feature that could potentially see over-use and ab-use. But then, so did templates :-P
And clearly someone who had never once written code for a system without a filesystem.
I heard about #embed, but I didn't hear about std::embed before. After looking at the proposal, to me it does look a lot better than #embed, because reading binary data and converting it to text, only to then convert it to binary again seems needlessly complex and wasteful. I also don't like that it extends the preprocessor, when IMHO the preprocessor should at worst be left as is, and at best be slowly deprecated in favour of features which compose well with C proper.
Going beyond the gut reaction and moving on to hard data, as you can expect from this design, std::embed of course is faster during compilation than #embed for bigger files (comparable for moderately-sized files, and a bit slower for tiny files).
I'm not a huge fan of C++, but the fact that C++ removed trigraphs in C++17 and that it's generally adding features replacing the preprocessor scores a point with me.
[1]: <https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2020/p10...>
int main () {
return
#embed "file.bin"
;
}
That's a good point. I consider myself debunked.
People that don't like it generally just don't know how to use it.
#ifndef asdf
}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}
#endif
There are better tools for the functionality the C preprocessor attempts to provide. Other languages have module inclusion systems and very powerful macros that don’t have the enormous footguns of the C preprocessor.
Édit: to be clear, I think #embed is a fine idea; I’d use it and it would make my sourcebase cleaner in some places.
include_bytes! gives you a &'static [u8; N] which for non-Rust programmers means we're making a fixed size array (the size of your file) full of unsigned 8-bit integers (ie bytes) which lives for the life of the program, and we get an immutable reference to it. Rust's arrays know how big they are (so we can ask, now or later) but cannot grow.
#embed gets you a bunch of integers. The as-if rule means your compiler is likely to notice if what you're actually doing is putting those integers into an array of unsigned 8-bit integers and just stick all the file bytes in the array, short cutting what you wrote, but you could reasonably do other things, especially with smaller files.
That does sound soul-crushing. Congrats on this achievement!
We exist only as long as we are trusted to be good stewards, and only go forward with the consensus of the wider community.
It's amazing that you and the ISO team are good stewards of the C standard. Thank you for being part of that.
And it can also be true that it was "hell" and "hardly worth it" for the OP to get a new feature added to the language. I believe it was a miserable experience that has him questioning how he spends his time.
Both can be true. Thank you for your efforts. And thank the OP for his efforts too.
> This is simply wrong. We (the ISO wg14) don't hold the cards, compilers are free to implement what ever they want, users are free to use what ever tools or languages they want.
This is an incredibly oblivious realization of JeanHeyd's point.
That "standard" card seem to be a pretty huge one though.
George Bernard Shaw
edit:
My goal is to avoid reloading the page for each selection and rendering all items eagerly. JS frameworks are the only ones that really allow this behavior.
It was actually pretty neat, because you could have an HTML file with a template, style, and script section.
Safari rejected the proposal, so it had to get dropped.
But ESM makes it a bit redundant anyway. The end-goal is to allow you to import any kind of asset, not just JS. There have been demos and examples of tools supporting this going back over half a decade at this point.
<!--#include virtual="/cgi-bin/example.cgi?argument=value" -->
<?php require("somemodule.html"); ?> <calc> /* ... compute and return dom element */ </calc>
in HTML. Or maybe it's been discussed but got left out
<script>document.write(`<p>foo</p>`)</script>
<dom-calc>
const p = document.createElement('button');
p.innerText = 'hi';
p.onclick = () => alert('hi');
return p;
</dom-calc>
class CalcElement extends HTMLElement {
connectedCallback() {
setTimeout(() => {
const fn = new Function(this.innerText);
this.replaceWith(fn());
}, 0);
}
}
customElements.define('dom-calc', CalcElement);It was a feature in Chrome 36-79 and there were working polyfills to make it work on other browsers.
It was actually a great feature and I used it extensively on an old project back then.
CanIUse: https://caniuse.com/imports
(Now obsolete) tutorial: https://www.sitepoint.com/introduction-html-imports-tutorial...
That said, to me this seems like a great addition to the language. It's very single-purpose in its usage (so it doesn't seem to add much conceptual complexity to the language) and it replaces something genuinely painful (arcane linker hacks). I'm very much looking forward to using this as I often make single-executable programs in C. The only thing that's unfortunate is I'm sure it'll take decades before proprietary embedded toolchains add support for this.
The first commandment of C is: 'writing a naive C compiler should be "reasonable" for a small team or even one individual'. That's getting harder and harder, longer and longer.
I did move from C being "the best compromise" to "the less worse compromise".
I wish we had a "C-like" language, which would kind of be a high-level assembler which: has no integer promotion or implicit casts, has compile-time/runtime casts (without the horrible c++ syntax), has sized primitive types (u64/s64,f32/f64,etc) at its core, has sized literals (42b,12w,123dw,2qw,etc), has no typedef/generic/volatile/restrict/etc well that sort of horrible things, has compile-time and runtime "const"s, and I am forgetting a lot.
From the main issues: the kernel gcc C dialect (roughly speaking, each linux release uses more gcc extensions). Aggressive optimizations can break some code (while programing some hardware for instance).
Maybe I should write assembly, expect RISC-V to be a success, and forget about all of this.
But like Lua, the base compiler is really small and simple and can be embedded. And it’s “pseudo-interpreted”: ultimately it’s an ahead-of-time language to support things like function declarations after references and proper type checking, but compiling unoptimized is practically instant and you can load new sources at runtime, start a REPL, and do everything else you can with an interpreted language. Now having a simple compiler with all these features may be impossible, so worse-case there is just a simple interpreter, a separate type-checker, and a separate performance-optimized JIT compiler (like Lua and LuaJIT).
Also like Lua and high-level assembly, debugging unoptimized is also really simple and direct. By default, there aren’t optimizations which elide variables, move instructions around, and otherwise clobber the data so the debugger loses information, not even tail-call optimization. Execution is so simple someone will create a reliable record-replay, time-travel debugger which is fast enough you could run it in production, and we can have true in-depth debugging.
Now that i’ve wrote all that I realize this is basically ML. But oCaml still has weird quirks (the object system), SML too honestly, and I doubt their compilers are small and simple enough to be embedded. So maybe a modern ML dialect with a few new features and none of the more confusing things which are in standard ML.
I've never used it but Nim does support some hot reloading as well [3]. It also has a real VM if you want to run user scripts and has a nice library for it [1]. Its not quite Lua flexible but for a generally compiled language its impressive.
Recently I made a wrapper to embed access to the Nim compilers macros at runtime [2]. It took 3-4 hours probably and still compiles in 10s of seconds despite building in a fair bit of the compiler! It was useful for making a code generator for a serializer format. Though I'm not sure its small enough to live on even beefy m4/m7 microcontrollers. Though I'm tempted to try.
1: https://github.com/beef331/nimscripter 2: https://github.com/elcritch/cdecl/blob/main/src/cdecl/compil... 3: https://nim-lang.org/docs/hcr.html 4: https://bellard.org/tcc/
Unsafe Rust code I think fits this model better than C does: it relies on sized primitive types, it has support for both wrapping and non-wrapping arithmetic rather than C's quite frankly odd rules here, it has no automatic implicit casts, it has no strict aliasing rules.
> It feels like I wasted a lot of my life achieving something so ridiculously basic that it’s almost laughable
Which makes me think I should never get involved with an ISO committee, not something I want done fast at least
The people on the C standards committee are gold bricks that say things like we won't add anything to the standard that isn't implemented in two or more compilers.
The compiler writers program in C++. And say things like if you want that feature that's a good reason to use C++ and stop using C. But if the standards committee adds it of course we will.
Also embed is exactly the feature that end uses would find very useful and compiler writers would not care about at all.
100% agreed. I've always viewed C as a "bootstrappable" language, in which it is relatively straightforward to write a working compiler (in a lower level language, likely Asm) which can then be used to bring up the rest of an environment. The preprocessor is actually a little more difficult in some respects to get completely correct, and arguably #embed belongs there, so it's debatable whether this feature is actually adding complexity to the core language.
Your wish for a "C-like" language sounds very much like B.
There is so much more to remove: 1 loop statement is enough, loop {}, enum should go away with the likes of typeof, etc.
I wonder if all that makes writing a naive "B+" compiler easier (time/complexity/size) than a plain C compiler. I stay humble since I know removing does not mean easier and faster all the time, the real complexity may be hidden somewhere else.
I guess the heated arguments here help me understand how it could have taken so long to get this standardised, though, so that’s something!
Congratulations and thank you to the OP for doing this, and thanks also for this really interesting (if depressing) view of the process.
One of the fundamental ideas that people run with in language creation/expansion is "programmer is stupid and/or make mistakes" -> "lets add language features that intercept and control his stupidity/mistakes".
And there is a very valid reason for this - it allows programmers of lesser skill and knowledge base to pick up codebases and develop safe software, which has economic advantages in being able to higher less experienced devs to write software at lower salary points and spend next to no time fixing segfault issues due to complex memory management. The whole reason Java got so popular over C++ was because of its GC - both C++ and Java supported fairly strong typing with classes, but C++ still had a lot of semantics around memory management that had to be taken care of, whereas with Java you just simply don't do anything.
However, people are applying this idea towards lower level languages, because they want the high performance of a compiled language with a whole bunch of features that make writing code as mistake free as possible. And my challenge to that is this - why not spend the time making just smarter compilers/tooling?
Think about a hypothetical case where Rust gets all the features added to it that people want, and is widely used as the main language over all others. Looking at all the code bases, there will be a lot of common use patterns, a lot of the safety code duplicated over and over in predictable patterns, e.t.c. And you will see these common things added to Rust. Just like with Java a lot of the predictable use patterns got abstracted into widely used libraries like Lombok, Spring, e.t.c, where you don't have to worry about correctness in lieu of using a library. And you essentially will start to move towards more and more stuff being handled for you automagically, which is all part of the compiler/toolchain
In the same way, #embed can be solved by smart compiler. Have a static string that opens a file, and read contents into a buffer that doesn't change? Auto include that file in a binary if you want to target performance rather than executable size. No need for special instruction, just be smart about how you handle an open call, and leave the fine tuning of this to specific compiler options.
And from an economic perspective of ease of use from above, you would have a language like Python which is super easy to pick up and program in, except instead of the interpreter, you would have a compiler that will spit out binaries. Python is already widely adopted primarily of how easy it is to set up and use. Now imagine if you had the option to run a super smart compiler that highlights any potential issues that come with dynamic typing because it understands what you are trying to do, fixes any that it can, and once everything is addressed, it spits out an optimized memory safe executable. With Rust, you code, compile, see you made a mistake somewhere with a reference, fix it, repeat. With this, you would code, compile, fix the mistake somewhere that the compiler warns you about, repeat. No difference.
Focusing on the toolchain also lets you think about integrating features from languages like Coq with provability, where you can focus not only on correctness processing/memory wise, but also "is the output actually correct". I.e, any piece of code for all given input can be specified to have guaranteed bounded output set, which you can integrate into IDE tools to provide you real time feedback on this for you to design the code in a way that avoids things like URL parsing mistakes, which all the languages safety features of Rust won't catch.
As for C, you leave it a version that has a stable, robust ABI, and then anything that you need to support will be delegated to custom tools. That way, in the future where compute will likely be full of specialized ML chips, instead of worrying about writing the frontend to support every feature, you quickly get a notional tool chain made and are able to run existing C code.
Just a random thought, but I'd expect a compiler to do exactly what's described if I tell it:
static char foo[123] = {
#embed </dev/urandom>
};
in embed_dump.cpp:
#include <fstream>
#include <iostream>
int main(){
std::ifstream f;
f.open("./source.png");
std::cout
<< "//automatically generated by embed_dump from project files:" << std::endl
<< "const char embedded_tex[] = {";
char a;
while(f.good()){
f.read(&a,1);
std::cout << int(a) << ",";
}
std::cout << "};" << std::endl;
f.close();
}
main_stuff: main_stuff.cpp embedded_files.h
c++ main_stuff.cpp
embeded_files.h: embed_dump source.png
./embed_dump > embeded_files.h
embed_dump: embed_dump.cpp
c++ embed_dump.cpp -o embed_dumpThis #embed feature might be a nice alternative for small files. Well for large files you usually don't even want to store them inside the binary, so the compilation overhead might be miniscule, since the files are, by intention, small.
When I read the introduction of the article - about allowing us to cram anything we want into the binary - I was hoping to see a standard way to disable optimizations (When the compiler deletes your code and you don't even notice).
More likely, the actual executable is only a small part of the file which accesses the rest of the file as an archive, like a self-extracting zip. There may also be some DRM trickery going on.
Although I've only seen 1GB+ pds, executables always in the hundreds.
It's a crutch, but at least you don't need to stuff the shader into multiple "strings" or have string continuations (\) at the end of every line. Plus you get some syntax highlighting from the embedding language. I.e. the shader is highlighted as C code, which for the most part seems to be close enough.
[1] https://github.com/phoboslab/pl_mpeg/blob/master/pl_mpeg_pla...
Thanks, I'll remember to use that in my future shaders
https://twitter.com/rcs/status/1550526425211584512
nullptr! auto! constexpr!
https://sentido-labs.com/en/library/cedro/202106171400/#bina...
That’s the same that `xxd` does, for instance, and it works with C89/C99/C11.
The advantage is that since it uses the same syntax as C23, it is easier to switch to using the compiler: just remove the Cedro pragma `#pragma Cedro 1.0 #embed` and it will compile as C23.
The source code (Apache 2.0) and GitHub link are at: https://sentido-labs.com/en/library/
ELF-based variants of the IAR toolchain, for example, provide a means of directly embedding a file as an ELF symbol, but without the size information being directly accessible.
GNU ld and LLVM lld do not provide any embedding functionality at all (as far as I can see). You would have to generate a custom object file with some generated C or ASM encoding the binary content.
MSVC link.exe doesn't support this either, but there is the "resource compiler" to embed binary bits and link them in so they can be retrieved at runtime.
Having a universal and portable mechanism which works everywhere will be a great benefit. I'll be using it for compiled or text shaders, compiled or text lua scripts, small graphics, fonts and all sorts.
[1] https://balau82.wordpress.com/2012/02/19/linking-a-binary-bl...
As an example, see [1]. That will turn any file into a C file with a C array, and I use it to embed a math library ([2]) into the executable so that the executable does not have to depend on an external file.
[1]: https://git.yzena.com/gavin/bc/src/branch/master/gen/strgen....
[2]: https://git.yzena.com/gavin/bc/src/branch/master/gen/lib.bc
Or source line location debug info, though nobody tries to show that for data at the moment.
One thing that isn't clear from skimming the article, how do you refer to the embedded data again?
I.e. you most likely use it to initialize a static variable, and then refer to that variable.
static const char d[]={
#embed ...
};
I am wondering, though - where does this stand in C++?
It will be useful to achieve it directly in the preprocessor however. I wonder how quickly can it be added to cpp?
It's also only suitable for tiny files: compile time and RAM requirements will blow up once you go beyond a couple of megabytes.
Do you know what makes it so? Is there a technical argument why the compiler could do better, except maybe for xxd not being specifically optimized for this use case?
It's allowed to do whatever it wants so long as the results are as-if it did what the standard says. So even though the standard says this is making a big list of integers like your xxd command, the compiler won't do that, because (as a C compiler) it knows perfectly well it would just parse those integers into bytes again, just like the ones it got out of the binary file. It knows the integers would all be valid (it made them) and fit in a byte (duh) and so it can skip the entire back-and-forth.
Touch grass indeed. Sure, #embed is a nice feature, but this self-indulgent writing style I can’t stand.
It belongs in the linker and you can pull the symbol it creates in with extern. I’ve been doing this for about 25 years.
If that's a problem, use Go or another higher level language.
I did. The author struggled embedding files into their executables with makefiles. We don't know anything else beyond that. So what?
People also struggle with memory management in C, an arguably much more difficult and widespread problem. Should we introduce a garbage collector into the C spec? How about we just pull in libsodium into the C standard library because people struggle with getting cryptography right?
OP mentions #embed was a multi-year long uphill battle, with a lot of convincing needed at every turn. That in itself is enough proof that people aren't in clear agreement over there being a single "right" solution. Hence, leave this task to bespoke build systems and be done with it. Let different build systems offer different solutions. Allow for different syntaxes, etc. Leave the core language lean.
It took a long time to get this adopted because people are most likely busy with things that cannot be already solved trivially.
The article covers quite a few reasons why the way things are done without #embed are not quite as trivial as they seem.
This proposal doesn't even allow to compress or encrypt the data.
While a noble act, this is nearly as inefficient as using a code generator tool to convert binary data into intermediate C source. Other routes to embed binary data don't force the compiler to churn through text bloat.
It would be much better if a new keyword were introduced that could let the backend fill in the data at link time.
According to the benchmarking data this extension is even 2x faster than using the linker `objcopy` to insert a binary at link time as you suggest.
Feel sorry for crab people.
C/C++ seems to be kicking it’s own ass.
> Surprisingly, despite this journey starting with C++ and WG21, the C Committee is the one that managed to get there first
Later it mentions presenting their first formal attempt at this to Belfast 2019, that's a C++ meeting, it's too late for this to go into C++ 20 at that point, but it easily could have been in C++ 23 (it is not).
But it's a bit boring. Unofficially, Rust has a mascot, in the form of a crab named "Ferris". The crab mascot appears in lots of places, and the Unicode crab emoji U+1F980 is often used by Rust programmers to indicate Rust in text. Unlike the trademarked logo, you can have a bit of fun with such an unofficial symbol, for example Jon Gjengset's "Rust for Rustaceans" book cover has a stylised crab wearing glasses with a laptop apparently addressing a large number of other crabs.
I can’t be the only one who thinks magic comment is already an ugly escape hatch, adding a mini DSL to it that can mean anything to anyone just makes it ten times worse. It’s neither beautiful nor great.
> do extensions on #embed to support different file modes, potentially reading from the network (with a timeout), and other shenanigans.
(Emphasis mine.) My god.
"#" is not a symbol for a comment line but the one for a pre-processor directive. Like #include stdlib.h
In c/c++ you use // and /* */ for comments.
cpp is definitely a well-hated part of C.
Please forgive those of us who've been using C since the 80's, or earlier, from assuming you don't know C when you invent your own terminology for preprocessor directives.
https://www.open-std.org/jtc1/sc22/wg14/www/docs/n3017.htm#a...
The major performance benefit is from the as-if rule. The compiler is entitled to do whatever it wants so long as the result is as-if it worked the way the standard describes.
So a decent compiler is going to notice that you're #embed-ing this in a byte array, and conclude it should just shovel the whole file into the array here. If it actually made the bytes into integers, and then parsed them, they would of course still fit in exactly one byte, because they're bytes, so it needn't worry about actually doing that which is expensive.
Does it work if you try to #embed a small file as parameters to a printf() ? Yeah, probably, go try it on Godbolt (this is an option there) but for the small file where that's viable we don't care about the performance benefit. It's just a nice trick.
If you want to include data from a continuous stream from a device node, then you could just as easily have the data piped into a temporary file of defined size and then #embed that. No need to have the compiler cater for a problem of your own making.
As for the custom data types. It's a byte array. Why not leave any structure you wish to impose on the byte array up to the user. They can cast it to whatever they like. Not sure why that's anything to do with the #embed functionality.
Both these things seem to be massive overthinking on the part of the committee members. I'm glad I'm not participating, and I really do thank the author for their efforts there. We've needed this for decades, and I'm glad it's got in even if those ridiculous extensions were the compromise needed to get it there.
yes, C finally catching up with what languages such as F# have been able to do for years with great success https://docs.microsoft.com/en-us/dotnet/fsharp/tutorials/typ... ; wild isn't it to step into the 2010-era of programming ?
?? pulling network resources works, today, with what people are using. there's zero difference between "cat foo.txt | xxd -i" and "curl https://my.api/foo.json | xxd -i"
These gratuitous standards prompt newbies to use the new features (it's "modern") and puzzled veterans to keep up and reinternalize understanding of new variants of languages they've been using for decades. There's no real improvement, just churn. Possibly it's one of the instruments of ageism. More incompatibility with existing software and existing programmers.
One problem of this new millennium is that the field has developed a tendency to do old things with new languages instead of doing new things with old languages.
Reinventing another polygonal wheel approximation while constantly tweaking the theory used serves to segregate the experienced (who may have trouble accepting such tweaks or their necessity - they already know about the existence of wheels anyway) from the newbies (who have no previous intuitions and no taste for legitimate and spurious novelty).
Newbies are cheap, and new ideas are hard. Let's do some mental rent seeking.