C Macro Reflection in Zig
jstrieb.github.io
jstrieb.github.io
For instance Odin allows to define a 'link_prefix' for C APIs which is then stripped from the imported symbols:
@(default_calling_convention="c", link_prefix="sg_")
sg_setup() => setup()
sg_shutdown() => shutdown()
> As a consolation prize, the TranslateC build step can be enhanced with advanced settings, such as namespace stripping and validation of existing bindings. Since changes to this don't require changing the language, it's OK if the scope & complexity increase to some extent.
Unfortunately, we must not forget forget ABOMINATIONCase and NAMESPACED_PascalCase (we could also generalize this to any switch from one convention to another after the first word). I've found they usually are, in fact, identifiable and roundtrippable. I've found the following usually works (it fails for multi-word prefixes or non-leading exceptions, and of course single-word identifiers are ambiguous):
count and strip all leading, then trailing, symbols (in some languages this is not limited to underscore but said languages usually need special handling anyway)
for every chunk separated by underscore, space, or hyphen (this may be nothing):
if the chunk either has no uppercase or has no lowercase, simply use it as a word. Otherwise:
for every sliding-window pair of letters (c, d) in the chunk:
if c is uppercase:
if d is lowercase, or d is a digit and there is lowercase elsewhere:
start a new word before c
space and kebab case don't have any good answer for affixes AFAIK. Otherwise, restore at least the leading underscores (trailing underscores are usually keyword-avoiding)
for snake and screaming case, be sure to prepend an underscore if the *result* would start with a digit
for camel variants, prepend an underscore to each *word* that starts with a digit. But if there were originally more than 1 leading underscores, use those instead for the first word. translateSpecialCaseNames(.{
.{ .src = "ABOMINATIONCase", .dst = "case" },
.{ .src = "NAMESPACED_PascalCase", .dst = "pascalCase" },
});
I do think that namespaces need to be semi-manually managed though (likely only at the project level), since often there are things that look like a namespace but shouldn't be treated like one, and it's not always obvious from a single identifier how many words should be treated as the namespace in some styles.
One special case is that sometimes C-ish libraries have class-likes like {Foo, FooBar, FooBaz} where the desired mapping is {foo.Foo, foo.Bar, foo.Baz}.
One of the main reasons I took Zig seriously was their C interop story—as someone who loves C and dislikes almost every implementation of C interop and FFI I’ve used in other languages (Rust is a notable exception to this), I was pretty much sold on Zig when I was able to, in a total of < 3-ish hours, (1) implement a complete Zig wrapper over a C library I wrote without introducing any overhead, (2) build a Zig program using the wrapper, and (3) cross-compile the program, statically, producing single binaries for 5 different targets…Linux (amd64, arm64), macOS (amd64, arm64), and Windows (amd64), ALL from one laptop using a single Zig executable and some simple flags. This was C interop and productivity I’ve never experienced before.
I respect Andrew a lot (I think he’s pretty brilliant), so I hope this turns out to not be as bad as I think it’ll be for those of us who love Zig for what brings to the table for programmers who are biased toward C.
I think that's an acceptable compromise.
Especially for C libraries which require configuration via preprocessor defines or compilation flags, the build.zig way is a lot cleaner than the @-builtins that are currently used (IMHO).
This does not seem bad in paper. One thing that does worsen with this is that in @cImport you could also comptime define preprocessor macros which was really cool, now that would have to be handled by build.zig I guess.
I’ve been able to convince C programmers to try out and use Zig just due to this unique ability alone, and to be clear, getting C programmers to seriously consider any language other than C is generally very difficult!
Having to consider another build system (with its own semantics), which the current Zig experience doesn’t require, changes the experience much more substantially than I think the Zig team realizes.
The big difference to the Rust ecosystem (or rather the 'cc' crates.io package which AFAIK is the current standard solution) is that there will be a Clang toolchain package with integrated cross-compilation headers and libraries that is used across all platforms instead of relying on a "platform C/C++ compiler toolchain" - which is the actually brittle part: different compilers, linkers and platform SDKs used on different platforms).
Ideally that same integrated Clang toolchain package used and provided by Zig could also be used by Rust to improve the C/C++/ObjC cross-compilation situation (similar to how the Zig toolchain is sometimes already used for this purpose).
Yes, this is a very good point. Zig remains unique and impressive in that sense. The fact that Zig compiles the correct libc and other system libraries on demand is actually another one of those, “How come no other language considered doing this before Zig?!?”
https://crates.io/crates/cargo-zigbuild exists though.
With hindsight, it is strange that the C community, with all the people and money behind it (and what scale!) never even managed to build a proper packaging manager (okay, there's now Conan, but that came from Python guys).
But then, there even still isn't a perfect C string library around (something that would combine GLib, SDS, ICU, say, and then standardize it in C2038).
[1] Hanson's C: Interfaces & Implementations (CII) - Str: https://cii.s3.amazonaws.com/book/pdf/quickref.pdf
[2] ICU - https://icu.unicode.org
[3] SDS - https://github.com/antirez/sds
[4] GLib - https://docs.gtk.org/glib/struct.String.html
The actual build system functionality is 'just' an extension to the stdlib.
So is the mess of ISO defined languages, where compiler toolchains are an abstract concept.
Also a good point to note that OpenGroup and POSIX never bothered to standardise UNIX package management.
It can compile almost any dependency on demand, even doing cross-compilation. And it's used by at least Microsoft and Google.
In Zig all those things are in the same toolchain install, which is a a single executable plus a bunch of platform-system-headers and -libraries, all under a single version number and trivially downloadable as a zip archive which works right after unzipping without setting up paths or running an 'installer'.
Package managers are a lot more complex than people realize. Every attempt I've seen in every language has significant lacks for common real world cases.
Most commonly they assume all the world is their language. Very commonly they want to take over all package management but don't have a good easy story for working with the OS package manager (I know Windows doesn't really have a package manager - but you still need to interoperate with it!). Those are big issues anyone in a complex project will face in the real world, there are also lots of little warts that will bite you.
Yes package managers are nice on your trival project. However they all get in the way in your non-complex project - some more than others.
Successful I agree. I have issues on any system I use it on because now I have the package installed by PIP and the package installed by my OS package manager - slightly different.
The anti-solution of ignoring the problem is what got us here.
Because if they don't integrate you end up with several different versions of the same package installed. When you program grabs the wrong version how do you fix it. Now what if you are in support trying to help a customer.
> on Linux, the distributed packages are normally about 2 to 4 years out of date
Maybe you need to find a different distribution. Some make it a point to be out of date. Some make it a point to be up to date.
https://github.com/ziglang/zig/issues/16270#issuecomment-161... https://github.com/ziglang/zig/issues/16270#issuecomment-161...
> This is not the first controversial change I have made to the Zig project, and it won't be the last. I have a vision, and I know how to execute it. People are often surprised by what Zig has accomplished, and they wonder why other projects have not done what Zig does. Well, you are seeing the magic right now. I ignore the peanut gallery and do what I know is right, while taking care of my users' needs at the same time. If you don't understand now, you will understand once this plan unfolds. By that time it will seem obvious in hindsight.
He's probably brillant and all but this ... feels like hubris.
In a world of humble bragging and cheap talk, I find it refreshing.
Plenty has already been said about design-by-committee and relying overmuch on user feedback, but one thing that doesn't get mentioned as often is that this approach tends to transform additive bias[1] from a cognitive bias into an iron law. Once you let that happen, you're on a relatively short and slippery slope to having a kitchen sink language. And one refreshing thing about Zig is that's it's clearly working very hard at not becoming a kitchen sink language. I'm not sure I can say the same about most other newer systems programming languages.
That doesn't mean having a BDFL is all kittens and rainbows, and I'm sure Andrew has made plenty of mistakes. But I've also never seen any indication that he's acted out of anything other than good faith. That last paragraph is possibly the closest I've ever seen to him saying something arrogant, and I see it as the exception that proves the rule. Finding such a tactful way to remind people that this is a BDFL project and he's the BDFL could not have been easy, and I imagine he put a lot of care into crafting that paragraph.
1: https://www.scientificamerican.com/article/our-brain-typical...
Rust definitely swayed more to the former than I'd have liked, but you also have Go as a counterexample where generics were dismissed for a decade+ in response to user feedback and then kind of :shrug: ok fine we'll add generics.
As someone who often thinks several steps ahead about where I want to take a project, I find it’s just as often difficult to communicate that vision at the level of detail necessary to establish a shared understanding of what those steps mean, and how various apparent minutiae come together to make them all valuable together.
I would be lying if I said I don’t wish I shared this particular hubris, and the corresponding expectation that execution will be sufficient to bolster any trust challenged along the way.
I don’t think it’s hubris, because Andrew’s results speak for him, but it’s certainly alienating, to be honest.
Although I understand the struggle of having to prioritize opinions and perspectives, it comes across like Andrew only values the opinions and perspectives of very specific folks, whom he often calls out.
Here’s what I mean: I love Zig and I write a lot of Zig code, especially within the past year (almost daily), but none of the Zig code I’ve been working on is publicly available or open-source (although I hope I can open-source various components soon, fingers crossed). I’ve gained a lot of valuable experience with Zig—including successfully convincing folks (mainly C programmers) to use it who wouldn’t have tried it otherwise. When I read these interactions, even though I have thoughts I’d like to share as a committed user who wants to see the project succeed and gain mainstream adoption, I get the feeling that my thoughts aren’t welcome since I don’t have a huge Zig project or something, so I just keep my thoughts to myself. Andrew seems to mostly care about feedback from the creators of Bun, TigerBeetle, etc., which, if I’m correct, is fine (it’s his project and therefore his right), but I imagine there are plenty of users like me who aren’t just part of “the peanut gallery” yet staying out of it to avoid the drama.
Don't get me wrong - I wish the best for both languages and am thoroughly impressed by the work of their creators. I can see that it must be a hard thing to balance.
What he should have done is announce a project at the same time that will maintain the current developer experience, even if that project is not part of the zig foundation. The developer doesn't care about how he builds zig. They want 1) download 2) use. It doesn't matter from where. If today it's directly from zig, and tomorrow it's from elsewhere to download a combined package, that 's all the devs needed to hear.
Weird machinations around projects that are and aren't (but are privileged because he's the creator) part of the zig foundation would be more concerning, quite frankly.
Unrelated to Zig, it's the exact same thing with "WASM can't access the DOM" btw, comes up everytime when WASM is in the news, but is a complete non-issue in practice for anybody actually using WASM, and I'm sure each popular software project has at least one such 'drama issues'.
The 'LLVM divorce' has been announced waaaay ahead (maybe years) of any actual steps to make that happen, for a language ecosystem that's still unstable anyway, and with the promise that a solution will be in place before LLVM is actually kicked out.
Not sure what else could have been done better, and as other have said, this sort of decision making process is much preferable to a committee approach.
LLVM is not and was never going away as a compiler backend, it will just not be the default one and you will be able to compile zig without it (though in practice everyone will for prod releases)
In this case, this is not a non-issue. If you look at the original github issue, it does a lot of damage, followed by a lot of confusion, followed by a damage-control post that shouldn't have been required if it the original issue was written with more care. It wasn't, because the marketing aspect of Zig was not the focus -- the technical issue was. So it blew up. Hopefully it'll be a lesson learned, but I suspect it isn't. It will take 20 more of such incidents before it sinks in.
As far as it was "announced years ago", I don't see how that matters. The people who are seeing the issue now, and this discussion weren't there to see the announcement years ago.
But I do understand what you're saying. You're point is that the onus is on the the reader to the do their research before overreacting. That's a perfectly fine position. I have a counter opinion which is that the person making the statement / claim / annoucement, simply be understanding of the implication of their statements.
This thing blowing up should not have been a surprise to anyone. Sounds like it would have been to you, so the fact that it blew up is evidence that you would have misjudged. Unless you also agree that the initial message should could have been better worded, in which case, what exactly are you disagreeing with?
Have you tried D? It can import C files as if they were D modules: https://dlang.org/spec/importc.html
Basically, if there's a `hello.c` file next to your D file, you simply import it with:
import hello
import hello: square;
import hi = hello;
C libraries (header files) can be compiled to D and then used as D modules as well, see https://github.com/jacob-carlborg/dstep
Is that as good as Rust?
I tried D years ago (at least 5 years ago, I think), but I don’t remember experimenting with the C interop.
Your explanation sounds intriguing though. And a couple pretty smart folks I respect have talked about D too. So thanks for bringing it up. I’m going to make some time to experiment with it again some time soon.
> Is that as good as Rust?
Franky, although I don’t have firsthand experience with D, your explanation and example make the D C interop experience sound actually better than Rust’s. It seems more similar to Zig.
As it is currently described, @import of C will require build.zig and specific configuration therein.
I consider that an appropriate development. Binaries need to be built, this should be integrated with the rest of the system in any modern language.
It's not incompatible with using something like CMake or Ninja either, it just puts certain responsibilities inside of build.zig. Where they belong.
const translate_c = b.addTranslateC(.{
.root_source_file = b.path("src/c.h"),
.target = target,
.optimize = optimize,
});
exe.root_module.addImport("c", translate_c.createModule());
[1] https://github.com/ziglang/zig/issues/20630#issuecomment-225...
You’re oversimplifying what’s actually a significant change.
Right now, I can do the following:
1. Download a prebuilt Zig release [0], on a fresh computer (zero prerequisites).
2. Implement a C library foo via foo.h and foo.c.
3. Compile foo.c into an object file foo.o via `zig cc`, using standard GNU-compatible compiler flags, for virtually any of the common (and even not-so-common) compilation targets.
4. Implement a Zig program bar via bar.zig.
5. Directly import foo.h into bar.zig via `@cImport` and use the C API as if it was Zig code.
6. Compile bar.zig and foo.o into a statically-linked executable baz via `zig build-exe`, for virtually any of the common (and even not-so-common) compilation targets.
No knowledge of the Zig build system and its semantics necessary. This is one of the most attractive aspects of Zig for a C programmer. I’ve gone through these exact steps with C programmers who thought Zig seemed interesting in theory but just didn’t want to have to learn a new ecosystem, build system, etc. The moment I showed them how quickly they could get up and running with Zig (and test it out with C code), without even having to install anything, it suddenly was impressive enough to experiment with.
The build system requirement may seem minor if you’re already a Zig programmer, but it’s massive if you want to attract C systems programmers.
You won't need knowledge of the build system nor a 'build.zig'.
After step 4 you would run 'zig translate-c' on 'foo.h'. Then use '@import' instead of '@cImport' in step 5 for the translated file.
'@cImport' is basically just doing that under the hood. It's an additional step for the user, that's a fair point, but I definitely wouldn't call it massive.
So `zig cc` will have to go as well ? I was under the impression Zig as a drop-in C (cross-)compiler was one of its main selling points.
The ability to seamlessly compile C/C++/ObjC code within a Zig project is extremely important for me as well, but I'm fine with that job being delegated to a Zig package and the Zig build system.
I mean, we'll have to wait and see, and there may be a transition period during which things kinda suck, but zig has a pretty good track record here so I'm more optimistic than I would be in most cases.
What's happening is that C imports are moving to the build system, instead of being a compiler builtin. It's part of making LLVM and libclang optional for programs which don't use it, but the use case of building C programs, and integrating C libraries with Zig programs, remains a central design goal.
The build system is relatively new, and a lot of things which were originally independent subcommands are being consolidated into the build system.
There's a sort of ambient impression that "Zig won't support C anymore" floating around, I'm not sure from your post whether you have that impression or not, but it isn't even vaguely true.
It just means that importing C libraries will be a build step, and not something you write directly into the relevant source code. This is not a big deal.
It 100% is a big deal. I explained this in another comment [0].
No, I’m not. Respectfully, you’re responding to my point while admitting in your comment [0] that you don’t actually know.
If you read the relevant discussions, the conclusion, last time I checked, was that there’s going to be a build system requirement.
You’re making a lot of “assumptions” when you could just read the GitHub issue regarding this change, written by Andrew himself, titled “move @cImport to the build system” [0].
By the way, please note that I not only write Zig code almost daily, I’ve personally contributed to the Zig build system.
> Question: "So after this change, is there way I can still simply call zig run or do I have to use a build.zig file?"
Andrew's answer: "No, this use case will regress."
This in fact literally states that "just" calling "zig run" won't be possible anymore, and heavily implies you'll need a build.zig file.
> This in fact literally states that "just" calling "zig run" won't be possible anymore,
This seems correct.
> and heavily implies you'll need a build.zig file.
This seems to not be the case.
The idea that C programmers, C programmers, are going to start bailing en masse out of the top of the Zig funnel, because they have to copy-paste a build.zig gist to get started? This is risible.
The complexity has moved to a slightly different place. That's it.
What I find incredible about these takes is that they’re so removed from reality. It’s quite unfortunate how difficult it is to get a lot of programmers to evaluate things from the shoes of others.
I have firsthand experience trying to get fellow C programmers to try Zig (and Rust too), and it’s already extremely difficult as is. The three things that have helped me sell Zig has been the combination of (1) how easy it is to get up and running with it (no prerequisites or installs necessary) [0], (2) the C interop story (which was a selling point for me [1]), and (3) the very simple yet impressive and complete build process [0].
What’s interesting is that you call my point risible when, based on your response, it’s clear to me you don’t understand the audience we’re talking about.
For example, you say:
> […] because they have to copy-paste a build.zig gist to get started […]
Professional C programmers generally favor actually understanding what’s happening under the hood, so unlike, e.g., JavaScript or Python developers, suggesting they get started by copying and pasting build code they don’t understand is a nonstarter for a lot of C programmers. And understanding Zig’s build system doesn’t happen as quickly [1] as being able to use Zig directly. Even after writing a decent amount of Zig build code, I’ve had to, on a number of occasions, go read the Zig build system code to understand how something is implemented (e.g., how it deals with certain paths, peculiarities of the dependency system, caching semantics, etc.)—if I had to figure out the build system before I was able to get started with Zig and be productive, I would’ve never taken it seriously.
There’s a reason why Zig has been marketed as a compiler that can also serve as a drop-in replacement for GCC/Clang. What you’re effectively saying is that this main selling point doesn’t matter even though that’s been one of the main things that makes Zig attractive to C programmers. Incredible.
Really, I think what C developers want is a modern, opinionated tooling solution a la cargo. If the Zig build system can deliver that (plus the much better ergonomics of the language itself), I think it could be very compelling.
I’m sick of Make, CMake, Nix flakes, Docker, manually bundling specific arm cross toolchains, etc. If I can send someone a Zig + C project and the zig.build Just Works, I’m all for it.
- You're used to the way it works now, and don't like that it's changing
- The build system is woefully underdocumented (understandable, since it's new and hasn't stabilized, but it's not a good thing), and you're conflating that with the system being hard to understand, which it isn't.
- You're combining these things, and your inflated sense of how well you know every C programmer in existence, into a claim which, again, I consider risible.
Because you're saying things like this:
> There’s a reason why Zig has been marketed as a compiler that can also serve as a drop-in replacement for GCC/Clang. What you’re effectively saying is that this main selling point doesn’t matter even though that’s been one of the main things that makes Zig attractive to C programmers. Incredible.
Which is a bizarre thing to derive from what I said, and is also absurd. Andrew wants Zig to be the best way to build C, always has, and he's the one responsible for making it as good as it is. His goals there haven't changed, what's changed is how he intends to provide that.
Now, you can claim, and you have, that he's going to make it much worse in the process. But your reasoning for saying this is thin and I disagree with that premise completely.
const win32 = @cImport({
@cInclude("windows.h");
@cInclude("winuser.h");
});
const win32 = @import("win32");
const win32 = @cImport({
@cInclude("windows.h");
@cInclude("winuser.h");
});
pub fn main() !void {
_ = win32.MessageBoxA(null, "world!", "Hello", 0);
}
import windows, winuser;
void main() {
MessageBoxA(null, "world!", "Hello", 0);
}
Sometimes people ask for a special syntax for importing C files, but I like this simplicity so much better.
Naked imports are also annoying. Which import did that MessageBoxA come from? windows? or winuser? Is it in the language kernel?
Explicit is better than implicit. The utter pain for the code writer of four or five keystrokes here and there is not worth confounding the code reader.
> Which import did that MessageBoxA come from?
If it exists in two or more imports, the compiler will give an ambiguity error. To resolve the ambiguity error, qualify the call with the name of the import:
windows.MessageBoxA(null, "world!", "Hello", 0);
import windows : MessageBoxA;
import windows, winuser;
alias MessageBoxA = windows.MessageBoxA;
(it might be worth having some sort of lint option that will flag using the non-explicit form because I can see large multi-developer projects finding it preferable to add 'use the explicit form' to their local coding style ... but I don't have a dog in this fight so 'free thought, worth exactly what you paid' applies ;)
VSCode is great when it works, but in cases like these it quickly becomes obvious that it's a hodge-podge of tools glued together via extensions and command lines rather than a purpose-built IDE.
* https://marketplace.visualstudio.com/items?itemName=llvm-vs-...
I just prefer using Zig on emacs with the built-in eglot LSP client. You just tell it where you installed ZLS (and it won't do magic to get it for you) and off you go. It's the same level of support as with VS Code. If you're not new to emacs, maybe consider that (otherwise emacs may be too much of a rabbit hole :)).
If you use ZLS [0], make sure you’re always using the right version for the Zig version you have installed on your machine. In my experience, that fixes 90% of editor issues I’ve encountered using Zig (I don’t use Visual Studio Code though, so it’s possible your editor issues are related to the editor itself).
Nothing wrong with that, imo. It's hard to evolve a language without having the ability to make breaking changes. They're doing the right things, generally making the right calls, and taking their time building instead of rushing.
I really want to get started with Zig but I don't want to go back to the age of nano/edit.com for learning a new language. Zig is complex enough already.
You could hop on the Discord and get some help with your configuration if you'd like. All things Zig are somewhat underdocumented currently, that could use some polish for sure.
I recently started a new project. zig-init provides you with a working build.zig file and two very minimal project files under /src, one of which is a hello world binary and the other is a hello world library.
> Also I've had a lot of issues getting editor integration to work correctly.
ZLS is not at the same level of what you would expect from a more mature language. For example by default it will not help you with anything related to comptime.
I highly recommend reading this fairly recent blog post by Loris Cro:
https://kristoff.it/blog/improving-your-zls-experience/
It explains how you set up your build.zig and ZLS (a few lines of configuration) in order to get much more help from ZLS. It will then perform the equivalent of what `cargo check` (Rust) does.
I think its fair criticism that as a beginner, you don't actually want to deal with build.zig or even understand it. You just want to write some code and run it and look at build.zig after you gained some familiarity with the language itself.
So there is nothing technically impossible about having the access to macro names as a compiler-specific extension, it's just that there is no much demand for it.
It's not probably reflected to the language because C++ fears reflection in general and hates macros in particular.
Macros aren't a particular case of reflection... And at least in the way they're done in C++, they're a big source of bugs / spaghetti.
But only wicked gods would banish us from paradise
Why do they fear our power?
Because evil is what they are.
They got rid of "good concepts" because when attempts were made to implement them the build for a simple "hello world" was over 15 hours and the implementers didn't think they could optimize that enough to have acceptable build times. As such I wouldn't call what they got rid of good concepts - to be good it needs to be nice not just in an academic paper, but also in the real world.
fn stringFromMsg(umsg: c_int) [:0]const u8 {
@setEvalBranchQuota(1000000);
inline for (@typeInfo(win32).Struct.decls) |field| {
if (field.name.len >= 3 and std.mem.eql(u8, field.name[0..3], "WM_")) {
if (umsg == @field(win32, field.name)) {
return field.name;
}
}
}
unreachable; // umsg is not valid, programming mistake
}
In zig-budoux, I also do comptime reflection on cImport struct to assert compile time that we won't produce broken runtime code
https://github.com/Cloudef/zig-budoux/blob/master/src/c.zig#...
IME there's is no difference in (optimized) code generation between an if-else chain, a switch or (like in your example) an unrolled for-loop with an if inside. All those high level constructs will be optimized into a single lookup table, or a combination of multiple lookup tables and binary search to select between those. Only if there are absolutely no consecutive ranges in the switch-set, a binary search without jump tables will be used.
But I also habitually run with jacked up contrast and nerfed brightness, so I make no claims that what works for me would be a good idea for anybody else.