Zig: All Package Management Functionality Moved from Compiler to Build System
ziglang.org
ziglang.org
Plenty people do it every year, naturally nothing to compare to GCC/clang.
I guess education system went downhill since then, which I guess it is kind of true, given the poliferation of header only libraries, as if C was a scripting language.
Rust language mistakes are understandable, because they do a lot of novel stuff so they encounter situations that no one have seen before. For the three languages that I mentioned, their mistakes are fairly well-known errors with obvious consequences. How hard was it to foresee that lacking generic in Go is a mistake, when the language was created well after Java had generics?
I found really cool that you have a bunch of options to configure compilation of source code itself. Not just the compiler optimization but you can automate all kinds of things: https://ziglang.org/learn/build-system/#build-system
Sadly, Gentoo is not great for managing per-project dependencies in the same way as is done by npm, pipenv, etc. Nix however works great (if you can stomach its stdlib).
I think that a programming language — much less a programming language _ecosystem_ — is such a large space of decisions that statistically you're inevitably going to let a few of them slip. And even (as in Rust's case, initially) if you don't aim to do anything new, the combination of the interactions between features and the ways people want to use your language culturally can land you places you never anticipated when doing the initial language design. I don't think anybody in the early years of Rust could have anticipated how async would look today, for example, and IIRC Graydon Hoare still doesn't like where it ended up.
Then, there's the old software adage that the right decision for one level of adoption doesn't necessarily translate to another level. Because it's so hard to predict the impact of early decisions, early versions of programming languages are basically prototypes: your aim is to get out enough of the core differentiators of the language that people can start playing with it and _imagining_ what it will be like to work with the final thing. Part of that involves making the barrier to entry as small as possible (e.g. bundling a full build system into the precompiled compiler executable) but it also implies that you don't want to spend more time than necessary on things that work the same as other languages. If you can see the ‘obvious omission’ in the language then that means you can already imagine how it will work when that thing is implemented! As an implementor you can always flesh it out as you approach 1.0, or, if the right thing to do is so obvious and you have an enthusiastic open-source community, you can just wait for the community to build it for you.
Go's generics case is a bit different, I think. I don't claim to be an expert here (I haven't followed Go development much) but as far as I understand it the omission of generics from Go was a deliberate ideological choice: they hoped to get by with the absolute minimum of generic functionality, and the experiment was how little they could get away with and still have it adopted. (Unfortunately, I think the experiments Go was trying to run were stymied by Google and Kubernetes throwing their weight behind it, which led to a pattern of adoption that had little to do with the language design itself: we may never know what would have become of it if it had been left to stand on its own.)
Most performance bottlenecks aren't inherent limitations of the hardware or the problem. There's some minimum amount of effort required to get the result, but that's never the reason why the application doesn't perform well. It's always the cruft we add on top.
It happens in many projects .. devs will just ‘do something convenient so they dont have to do the work to do it properly’ and then that just becomes the convention, because nobody else wants to do hard work to do things properly, either.
Which is why it’s so frustrating when bikeshedders use indecipherable shades of paint.
Good code is best made understood at scale .. or at least, that’s how it used to be.
Anyway I think the Zig folks are pretty clever and know what they’re doing .. I”m sure there are holes to poke in all of it, but I wonder what the big Zig projects will do with this change…
You've basically arbitrarily focused on a particular flawed design decision of a language while ignoring a thousand others that they may well have gotten right.
It's not possible to just sit down and get everything 100% right up front. The language design space has too much both depth and breadth to it.
IMHO, you may be confusing “care” for legibility. The creator(s) made a choice that you disagree with and that you may not understand. They also haven’t justified or explained their choice to you. That illegibility might feel careless, even a little mean, but I reckon their focus was on figuring out the language, not seeking widespread approval.
If you wanted to implement a package manager in the build system and keep it in the compiler, what abstraction for it would you provide between the compiler and the build system? This probably affects how you design the import system, and some other fundamental parts of your language. Those are important to get right, and probably hard you change. If you start by tightly coupling the package management and the compiler, it could be easier to design that by starting with the programmer experience you want and working backwards. Then you can change the implementation later, affecting fewer users.
I'm thinking specifically of Python here, where the import system is crazily complicated and warty, partly to accommodate all the varied ways package management was done. This compares to Go where there is a comparatively neat and small syntax for imports and the like, because they were designed with package management in mind from the start. (Not that I agree with all its design decisions.)
If you cherry pick, any position is possible to argue for.
What is C and error handling?
What is Java and value types?
What is C++ and modules?
Even though I use LLMs every day, and have to admit they're remarkably good at many classes of problem, I don't want a programming language built by an LLM. Every line of code in a programming language, every decision, every trade off, matters. A vibe-designed/vibe-coded programming language would be a disaster. I don't know how else to put it, and I've never seen any model produce code that would convince me otherwise (even Fable, which is, in fact, a notable improvement over the prior best models). The models don't want anything. They don't have meaningful opinions. They don't know what comfortable vs. uncomfortable feels like in a language (or in a GUI or CLI interface at sufficient levels of complexity).
You can't get a language like Zig out of an LLM without simply copying Zig, and even then it would be a copy that is worse. (I mean, I'm assuming "copy with an LLM" means, "have an LLM write the spec and another build the language to the spec", not literally `cp` the source tree.)
https://github.com/ChristofferGreen/PrimeStruct/blob/d4f1a8e...
1. That a human, even a brilliant human with a lot of experience, can sit down and one-shot a language spec for a new language that is actually good.
2. That an agent can produce code that is good enough for a programming language that intends to last for decades without exhaustive human review.
We might advance to the point where 2 is true, we're not there yet. We'll never make a better human that can one-shot a good programming language. It takes iteration, with a human in the loop. Zig has taken ten years to get to this point, and is still occasionally experiencing major refactors.
If you want to argue an LLM could potentially accelerate development by some amount, I would agree with you. How much it could accelerate it is debatable. And, I understand why the Zig folks have decided not to accelerate in that way. There is a cost to it. You lose the junior programmer pipeline, as your "good first bug" list gets chewed up by people using LLMs. You still have to exhaustively review the code for a critical path like a compiler and AI code is hard to review; it doesn't have a point to it. The model doesn't want anything, so it's not always clear where it's going. Code without clear intention, like prose without clear intention, is hard to read and hard to review. It's verbose and often makes weird assumptions.
If I, for some reason, needed to implement a tiny DSL for something, I would 100% do it with an LLM. If I, for some reason, were tasked with building the best programming language to replace C, I would not hand it to an LLM (though I would get help from an LLM, because I don't know how to build a programming language beyond the tiniest toy interpreter or compiler, I'd need to read and understand every line of code, and use it daily, for it to turn out good).
Plus, current-gen agents/LLMs can't implement a non-trivial language spec without significant hand-holding.
This is mostly about human preferences right. If software is just taken as the end product, does it matter what "feels" good and doesn't?
It's also an open source project, so the end product is as much the codebase as it is the binary releases.
I've interacted enough with largely computer generated codebases to see that ergonomics problems easily grow and accumulate when LLMs remove the burden of dealing with those problems from the developer. I've always considered my laziness an asset. Now I would qualify that by saying that laziness is an asset for as long as it compels you to keep things simple and easily understandable so that the cost of making changes (whether that's measured in human gray hairs or tokens) doesn't grow with every change.
We've heavily adopted LLM's, to the point where I'll often not touch any code and have a better and more maintainable codebase than if it had been written by any of our developers (myself included). I'm not sure it would be possible if our philosophy wasn't explicity, no abstractions and defensive programming, but when every function is it's own thing with runtime assertions and ways of dealing with corrupt states as soon as they happen. Well... the LLM can do that when you can.
That being said. I think it's not new that you can write computer software without computer science, until you can't. Where I see the challenge with LLM's and software development as a business is that a lot of software developers work(ed) that can be automated because they don't actually require computer science as such. Having setup company wide "apps" for Microsoft's cowork, I think any sort of semi-expert level office job is going to be in danger. I'm not a huge fan, and I was never hyped on AI but it's ridilous what you can do in the enterprise office space (and how easy it is) when you're married with Microsoft.
Now those systems have evolved to become orchestration workflows engines with agents and tools integrations, thus several scenarios that still required proper code, have now been reduced to agent instructions.
I think many in HN don't have a clue how little traditional coding some enterprise scenarios require nowadays, plugging SaaS products with such tools.
Being a backend dev is more administration than actual classical coding.
Having spent a lot of time in enterprise which is not related to software development (or IT at all) I'm pretty familiar with a lot of the low / no code or Robot Process Automation systems. Which always end up requiring specialized developers, except, you can't hire software developers for them so you end up with various consultants creating huge messes. Microsoft's Cowork is the first time I've seen a system which can actually directly replace people who aren't experts in their business domain. It requires a lot less than any other system I've seen before. You need someone who has global, intune and AI admin in the Microsoft ecosystem, who know enough about software development to evaluate an app to be safe and efficient and how to debug it, and a domain expert. Then you'll have something which "just works". It's scary really.
I had no domain knowledge of PowerBI like at all, and while I could fill a data architect role, I would have never been able to do anything beyond a semantic model in Fabric and even on the semantic model it's not like I've ever worked with DAX. Yet Cowork (with read access to the Fabric workspace) could guide me through the process of creating a powerBI report set which was good enough that the business wanted to use it with a few minor improvements... Five months ago, this would have been a 2 people job. Now I was basically a puppet getting told which button to click in the report editor. If Cowork had write access it wouldn't have needed me in the loop once the data was delivered to the datalake it shortcutted to.
For many kind of products, where selling software isn't the main business, companies will settle for approaches like that you describe, writing classical business software isn't what it used to be.
We are at the evolution of 4GLs and RUP dreams.
For what you were asking though, it's not no-abstractions it's YAGNI where business logic lives in "just" functions which operate on plain data structures with runtime assertions and validation. The data structures will usually include domain specific types. There are no classes, object hierachies or architecture abstractions, at least, there almost never ever are. When it absolutely doesn't make sense not to introduce an abstraction beyond what we typically want, then it's allowed. This pains a lot of Uncle Bob's disciples, until it doesn't.
The advantage of it is that we've found that it works really well with LLM's. At least in my experience they seem to be absolutely excellent with explicit, localized code instead of layers of indirection.
I'd love to read a blog post or something on what you've discovered on good software patterns for AI. I'm only now getting into using LLMs and beyond the purely technical aspect of what LLM is "best" it seems like there's a wealth of learned experience in how to structure and write code to work well with them.
I've heard good arguments for using Rust with LLMs because the compiler keeps the LLM from making silly mistakes. What languages have you been developing in?
I think this is a bad move since the real fix to these attacks is a sandboxed environment rather than a single tool implementing sandboxing.
I don't know Zig's plan, but once you have the ability to broker privileges like this you have the ability to audit the privileges being brokered and things change forever.
In theory, you could run the whole compiler (including C) in WASM as well but I don’t think that’s the goal? You kinda need to trust the compiler itself.
The benefit of "whole environment" is that if you stuff everything into that environment then anything in it is confined, but it's all confined with everything stuffed in and is sort of maximally capable. You can rarely do things are significant as, say, system call filtering, because all software in the environment must continue to work and none of it was designed with that in mind.
Native sandboxing like this will likely make auditing much easier as well. If a dependency requires something like "give me the ability to execute code on the OS", now it has to ask for it and now it gets additional scrutiny.
Native sandboxing is and always will be the infinitely superior method when it's actually used. Whole process/ Environment is only what we use because of how rare native sandboxing is.
But when looking at open source code you don't trust yet, you might want to build code, without running it, so your development tools will work.
I added WASM/WASI bindings and wrote the prelude in Haskell.
Both the bindings and the prelude took less time than I already wasted on multiple attempts in Starlark that all collapsed into string goop.
It's nice to be able to bound the execution environment in a build tool but still use a serious programming language.
Given Zig has excellent support for targeting WASM, seems you'd get the same advantages.
Language-specific build facilities, like Cargo's build.rs and Zig's build scripts, typically have a limited scope - generating a bit of source code, discovering some linker flags, stuff like that. These scripts need to be run by LSP servers when opening the project in an editor to get basic features working, so that's a fairly risky thing.
They are also currently doing things like invoking CMake and other build systems, but you could definitely conceive of a world where that was a separate step in the build process, and that world seems pretty attractive to me.
A common pattern in Rust projects is to have a `*-sys` crate representing the C FFI bindings, and they typically also do something like invoke CMake or similar to actually build the C/C++ library underneath. But if you have a larger project that already integrates multiple build systems, this is really quite inconvenient in most cases.
There's still the obvious problem that if the build system emits malicious code, you'll probably run that code anyway. Personally I think this kind of sandboxing is more useful for enforcing build reproducibility rather than, like, protecting you from viruses in the build.zig file.
I know that it’s purely a UX concern, and that the changes (to decouple the build system and the compiler) are pretty critical for the maintainers, but it’s still a bit sad that development sanity comes first than the UX. (It’s the right call, just that it’s sad.) @cImport was a big killing feature imho to the language…
The only exception is C/C++, where there is none established that well, for good or bad.
These choices may create later super-convoluted processes when you have to mix more than one language together.
Packaging systems makes thing easy, but complicate further the line if another language needs to be used.
The only real such build systems are Buck and Bazel. But they have way too much baggage from their overlords.
It’s a shame.
A good polyglot build system should support llvm and zig and python and literally any toolchain under the sun.
no! No you do not!!
It’s perfectly totally fine to have multiple different compilers. Literally not a problem at all. That’s my point!
Compilers and build systems are separate. It is perfectly fine for one build system to invoke multiple different compilers.
Buck and Bazel have the right architecture. Just need a version written fresh without all the baggage.
Fixing this is beyond any one tool, of course :)
I've worked on many large projects in C++, and every single one of them contains a bespoke, buggy, undermaintained JSON parser, URL parser, configuration file parser, async framework, and so on. It used to be the case that almost every large C++ project started out by defining its own friggin' string type.
Dependency anxiety is a variant of NIH syndrome, and it leads to much, much worse quality software in the average case. Most companies are not in the business of writing a bug-free async framework, and yet here we are. The cost of vetting your dependencies is much, much lower than writing and maintaining all these things from scratch.
Having a standardized package manager allows lowering the bar and bypassing careful thinking. It has also a cumulative effect - if one adds each dependency in its project one by one with proper audits, transitive dependencies may not be managed so carefully. And then we have cargo-style cancer with trivial projects having hundreds of dependent packages.
I also think that languages should not have their own specific packaging system. This should be done on a distribution level, which provides curated lists of software. This system works well in the Linux world. The problem is the support for the commercial platforms.
The whole trend to think of programming languages as closed frameworks is bad. It is replacing a better system with one which is fundamentally much worse, instead of trying to fix the problems with the existing systems (which is hard because it needs collaboration and harmonization, instead of going into your own little bubble and pretending to do something superior).
I don't think that has been shown to be true. There's a lot of FUD, though. Tools like `cargo vet` and `cargo audit` seem to be pretty good.
> I also think that languages should not have their own specific packaging system. This should be done on a distribution level, which provides curated lists of software. This system works well in the Linux world. The problem is the support for the commercial platforms.
I mean, it works until it doesn't. There are really significant drawbacks to this as well, including the knack some distributors have for thinking they know better than the original developer.
The Linux distribution model worked perfectly fine for the 30 years I am using Linux.
What attacks are you referring to specifically?
> It is also obvious that this way to distribute software is susceptible to supply chain attacks. cargo audit is an attempt to fix this retrospectively, but does not fix this fundamentally broken model.
This is the FUD I'm talking about. If you worry about supply chain attacks, it's not harder to be careful just because dealing with dependencies in general is much easier. You have to vet your dependencies no matter what.
> The Linux distribution model worked perfectly fine for the 30 years I am using Linux.
The xz scandal was a damn close call. Linux is a forest of distributions, and each distribution is a separate vulnerable organization.
It is not FUD as it is a real risk. cargo audit can only search for known issues, as such is like a virus scanner, often being too late. "You have to vet your dependencies no matter what" sounds nice, but Rust packages tend to pull in many hundreds of dependencies. But even for the comparable fewer dependencies in other languages, you can not usually vet everything yourself, you need to rely on the community to help. But then you need to have a curated software distribution, and this is exactly what traditional Linux distribution model does, while cargo does not.
xz scandal was a close call, but it was also a very sophisticated operation - which was then discovered by a Debian developer. This is not at all comparable to the cheap type squatting style attacks possible with cargo and confirms my point.
Probably the one security sin of most language package managers is allowing anyone to upload to the central repository without review. This is good for accessibility but bad for security. There are tools like cargo-crev, though, which can help you enforce some level of vetting if you wish.
The actual problem is the complexity coming from the choice to use these things at all (and thinking they should play any important role in the system at all, instead of being entirely replaceable), which means to do anything useful you have to go through a lot of abstraction, which makes everything tedious and buggy and hard to fix.
It is as standard as in many languages where package management came after years of using the language.
OK, here's a few (with a MacOS slant): - compilers (gcc, clang, and their many versions) - libc (and friends) compatibility (I can't say I even ever delved into this one, but it's bit me) - package manager (macports, homebrew) - building for backward compatibility; what's the earliest MacOS version to support (the package managers either like to build for the OS they are running on, or force you -- yes I'm looking at you homebrew) - dependency and dependency version management (love you pkgconfig) - build system for each package (cmake, autotools, meson, ...) - bundling everything into an application - turning that application into a Mac application - code signing and notarization - creating the DMG - debug symbols - crash detection and notification Like right now, libheif on 26 can't be built for 11 and it's not clear why (or maybe they just fixed it...but it's been weeks)