Zig monthly, October 2021: Games, gamedev, Elixir, tools and more
zigmonthly.org
zigmonthly.org
I like Lisp and APL, so I have been learning April[2], a subset of APL in Lisp. It is very cool, if you like Lisp and APL. The game in Zig in the article reminds me of the Dyalog APL simulation of a boat navigation app developed in Finland using APL[3].
[0] https://github.com/scala-native/scala-native
[1] https://andrewkelley.me/post/zig-cc-powerful-drop-in-replace...
Go might be what you want. It's boring which means it just works with healthy tooling and libraries. I call it the true successor to C with a 1990s bell labs feel.
https://en.wikipedia.org/wiki/Ken_Thompson https://en.wikipedia.org/wiki/Rob_Pike
It's in a spot similar to Go, in that it doesn't have a VM, but is normally used with a garbage collector (the core language can do without it - the libraries are another matter, though). However, unlike Go, it doesn't adopt the "less features and more boilerplate is better" approach to PL design.
OTOH it does have templates (not just generics) every bit as powerful as C++ ones and then some - but also with fewer footguns, and overall easier and more pleasant to use.
You might also find it interesting that the D compiler has a mode that is specifically called "BetterC", which basically removes all features that are dependent on the D runtime library (GC etc): https://dlang.org/spec/betterc.html
It was very frustrating getting started though as just about anything Zig-related online you read/download/run is out-of-date with the latest version of the language. Small breaking changes go in that can easily stump a newbie like me, and there's not enough online experience of catching similar errors that give guidance.
IMHO I really hope the language stabilises soon. Otherwise it will be a frustrating experience.
Pre-1.0 semver is fairly meaningless, so taking it seriously really amounts to… doing anything at all. There’s literally nothing to be backwards compatible with, or to announce incompatibility with. They’ve have no opportunity to show how seriously they take it.
From the spec[0]:
> 4. Major version zero (0.y.z) is for initial development. Anything MAY change at any time. The public API SHOULD NOT be considered stable.
> 5. Version 1.0.0 defines the public API. The way in which the version number is incremented after this release is dependent on this public API and how it changes.
Which directly matches my expectations — the primary reason any project uses a version 0 (regardless of semver) is to specifically claim that they promise nothing — “no one should rely on this, and anyone who does is doing so at their own risk…”
Version 1 is I think near-universally agreed to mean “this is production-ready” — where production is vague but roughly implies “If it doesn’t do what it says on the tin, it’s a bug”.
Version 1+ meaning varies depending on the versioning scheme, but 0 & 1 are fairly universal.
Of course, qualify everything with the standard “it’s open source so I don’t actually owe you jack shit”
Any promise/expectation of stability in 0.x.y is outside the scope of semver, and communally is on you. Bevy for example breaks their APIs every 0.x release, and that’s perfectly fine & expected under semver.
In this scheme, 0.x.y is compatible with 0.x.z, so developing pre-1.0 software is as strict for breaking changes as post-1.0, just the numbers are different. For this very reason, many crates don’t rush into 1.0 territory and are comfortable in pre-1.0 zone.
The real test is whether they bump it to v2 or withhold an important change because of semver violation.
Here's a direct link to the ElixirConf talk:
https://www.youtube.com/watch?v=lDfjdGva3NE&list=PLqj39LCvnO...
I thought it was some sort of Erlang native compiler written in Zig (which sounds like an incredible pain in the ass), but it's really "just" a cross-platform installer. Still useful !
Using Zig and Translate-C to understand weird C code
Created a new post for it here on HN: https://news.ycombinator.com/item?id=29092677
[1]: https://zig.news/sobeston/using-zig-and-translate-c-to-under...
I think there are a number of disenfranchised "close to the metal" programmers. C has stagnated and won't likely move much, C++(++++...) has reached bonkers level of complexity, D is... D who, and Rust is highly opinionated, which is a double edge sword.
Rust is complicated by its very strict memory safety. While that's nice, I don't think memory safety is game developers #1 priority. A crash is annoying but acceptable.
Zig also doesn't assume a global allocator anywhere (malloc/free), and I think game developers often like to use custom allocators for performance reasons.
There's more extreme situations like Super Mario Land 2 for the Game boy where it's possible for the game to present it's own memory on the screen and if you understand how it works you can do things like ask the game to execute the ending sequence [0]. At it's most extreme a series of TASes was presented in 2017 wherein multiple different hardware systems were linked together via RCEs in different games to make a VOIP streaming setup from a bunch of consoles that don't actually have internet connections [1].
I'm sure there's all kinds of reasons this kind of UB is undesirable, and could be hazardous in some environments, but people have also done some really cool stuff with it!
[0]: https://www.youtube.com/watch?v=faiZgO35YaQ&t=0s [1]: https://www.youtube.com/watch?v=7CgXvIuZR40
The only thing stagnant on WG14 is their regard towards providing safer string, vector types and enumerations.
It of course has to remain with us due to the insane amount of code already written with it and some tooling (and eg for verification certain subsets of C), but it is not any closer to metal than other system programming languages, the “standard lib” is full of foot guns, it has a very weak type system, it has no way of enforcing basically anything. Also, due to the its lack of expressivity it relies on text-based macros (and those I hate with a passion, as there is hardly anything worse than text-replacing source code with no knowledge of syntactic elements).
https://floooh.github.io/2018/06/02/one-year-of-c.html
...and "modern C" is often misunderstood by C++ people who only know the fork of C known as the "common C/C++ subset" (which is basically an outdated, non-standard dialect of C that's stuck in the mid-90s):
https://floooh.github.io/2019/09/27/modern-c-for-cpp-peeps.h...
However, I'm hoping to eventually abandon C in favour of Zig within the next 10 years or so :)
To me, C is the easiest way to interface with the platforms that I develop for. It's also totally sufficient to implement the ideas that I'm working on - figuring out the data layout, the flow of code and data... It offers a concise syntax for the frequent operations - load / store, arithmetic, dereferences, and function calls. Most languages are already disqualified by not offering or discouraging nesting of data types a.k.a value types, making it way too complicated to simply copy data "anonymously", using memcpy() or similar.
Think about that - the language that people can't stop bitching about because it doesn't offer "generics" is actually the one that allows you to read and write data generically without a tremendous amount of complexity and/or slowness.
That's why my personal route is to learn ways to not need the type based polymorphism offered by more complicated languages, and to get along with just straight code that can push _any_ data payload (type) to the next endpoint - without requiring the multiplication of junk in the type system and/or in the compiled binary.
Macros aren't used that much but they would be missed a lot if they weren't there to help in some cases where we need to hack an abstraction over (lexical) syntax instead of data. I said above that you can't abstract over languages' syntax, and C macros don't really do that... They're a hack, and sometimes tremendously useful.
I wrote a firmware recently for low power microcontroller. No memory allocations in code. The language is simple and easy to read unless one is purposely trying to be fancy. It is very performant - the end result and compile steps as well. And there are megatons of C libs for microcontrollers. Would not dream of using anything else for this particular task
Sure let WG invest their efforts where needed but for my own case I do not care.
C is the lowest common denominator. It has good compilers that just work, with no drama or surprises. It's portable across CPUs and operating systems. It will continue to work for all Unix-like platforms long after trendier alternatives have turned to dust.
It has good code formatters and analyzers. It's an easy language to completely understand, and it's easy to read if I don't try to get to clever with macros or type tricks.
I don't have to remember whether ${language_feature} is Considered Harmful yet, like C++ or Rust where the current best practices constantly become Bad Code in favor of the next trendy feature.
The C ABI is the standard for libraries on Unix-likes, and I usually want to use a couple of special-purpose libraries. Calling libraries requires no wrappers, FFIs, or anything else.
Yes, C has limitations. Lots of them. But it also has an enduring momentum that's not going away any time soon.
It has and it has not. It can offer level of complexity matching that of a programmer. One does not have to start coding concepts as the introductory course to programming. Take some numbers stuff them into array, sort and print - is a simple piece of cake anyone can grasp.
[0]: https://zig.news/kristoff/struct-of-arrays-soa-in-zig-easy-i...
That's just my memory though, nothing 'official'.
Also, most gamedev middleware is written in C++ or C, and Zig can build those out of the box and directly import C headers so it's trivial to build mixed Zig/C/C++/ObjC projects (and without the hassle of C/C++ build systems).
Rust is trying to solve problems we don't have. Jai is too closed.
Great that your customers are on the safe side.
https://www.bleepingcomputer.com/news/security/cs-go-valve-s...
Ironically enough this is the one class of bugs that Rust would not have prevented out of the box since the language defines integers to be wrapping and you need to manually opt into runtime checks.
The exploit:
https://secret.club/2021/04/20/source-engine-rce-invite.html
Wrapping in Rust:
https://huonw.github.io/blog/2016/04/myths-and-legends-about...
https://scattered-thoughts.net/writing/how-safe-is-zig/
I hope that explains how inappropriate is the tone of your comment.
What is the solution for use-after-free on Zig today, that isn't just like using Purify?
In other words, if you mean that use-after-free is handled in Zig in a manner that's very different from how it's handled in either Java or Rust and more similar to the general approach of how it's handled by various tools for C, then that's correct. If you mean that that general approach is just as (in)effective in Zig as it is in C, then that is incorrect. Because Zig is fundamentally more precise than C, analyses of Zig, even if they are conceptually similar to those done for C, can be more precise and effective.
Without pointer arithmetic, unsafe casts (Zig is much easier to write without such casts than C, at least), unsafe unions, and unknown buffer sizes, the set of pointers in a Zig program can be well-defined, as pointers can come into being only in very specific ways (they have a simple provenance). Because the set of pointers is well defined, it can be precisely tracked and analysed. This means that 1. pointers can be traced even with arena allocators or perhaps even other kinds of pools (provided they cooperate with the tool) and 2. dangling pointers can be detected even without being dereferenced. This is simply not something that analysers for C or C++ can do (at least not nearly as easily).
The way to think about it is that in Zig, when an object (including an allocator) is deallocated, you can invalidate the full set of pointers pointing to it, and that set it the only way of generating more pointers into it. That's not the case in C or C++. For one, pointers aren't well defined (unions); for another, a valid pointer could be used to create an invalid one.
So while the general approach -- of dynamic and static analysis, as opposed those of Java or Rust -- is in the same broad category of tools for C or C++, their effectiveness, due to Zig's properties, is significantly increased, and that their entire cost/benefit is different. I.e. it could find more bugs for a lower price, so much so that the approach, while underwhelming when applied to C or C++, could well compare favourably with others when applied to Zig.
(A GC -- whether tracing or ref-counting -- might well be adopted for Zig's comptime, but that's a whole other matter)
I'm highly skeptical that there won't be too many false positives with such a tool. Systems programmers routinely create temporary dangling pointers and let the values go dead without dereferencing those pointers. It happens most every time you call free, in fact.
I also see no reason why you couldn't create such a thing for C and C++. In fact, it exists: the Boehm GC can operate in such a checking mode. The fact that everyone uses ASan instead is a strong indicator that ASan is in fact a better approach.
Finally, precise tracing GC stack/register maps are a lot of work, especially in LLVM which has poor support for them. (I heavily looked into this for Rust.) Without a serious effort (and it is a lot of work) to generate them for Zig I have to consider it vaporware.
I'm not advocating for a specific algorithm. I merely used a hypothetical one to demonstrate that there is, indeed, a fundamental difference between Zig and C/C++, even when it comes to UAF.
> I also see no reason why you couldn't create such a thing for C and C++.
Because in C and C++ there is no precise set of pointers, and bad pointers can be created from good ones.
> In fact, it exists: the Boehm GC can operate in such a checking mode.
Which would not work effectively for the reasons I mentioned.
> The fact that everyone uses ASan instead is a strong indicator that ASan is in fact a better approach. ... I have to consider it vaporware.
So from your asserted premise that Zig is no different from C in this regard you conclude that what doesn't work well for C must not work well for Zig and use that conclusion as further evidence of your premise? Your premise is exactly what I contest. Zig is fundamentally different because pointers are known and you cannot manufacture bad pointers from good ones. To demonstrate that difference I sketched a hypothetical algorithm that could work well in Zig but not in C. To point out that my hypothetical example is "vapourware" completely misses the point.
You could try to argue that the fact that pointers in Zig are well-defined and can be created only in carefully controlled ways -- and so are fundamentally different from pointers in C -- cannot be effectively exploited, but I don't think that's an easy argument to make.
--------
(> especially in LLVM
Even if we were to talk about specific tools, there is absolutely no need to base them on LLVM. Zig is specifically designed so that backends are easy to write, and an analysis tool need not use the same backend used by the compiler for production code; in fact, switching backends is meant to be commonplace in Zig development, and it is expected that different ones will be used for development and production)
Zig does seem to have some properties that make precise pointer identification possible. But the right conclusion to draw from this is that Zig should use a tracing garbage collector. It's well-known how to use the pointer provenance properties you're talking about to achieve UAF protection: just implement a GC! Trying to get by with things like quarantining memory forever is not going to work in production, and the reasons why Zig programs will supposedly not be vulnerable to UAF problems are unconvincing. It is going to want a GC eventually.
I've repeated several times that I was merely demonstrating why Zig and C are fundamentally different when it comes to pointers. You're trying to poke holes at a straw man, and worse, you're doing that while drawing on the very premise which I'm refuting, i.e. that Zig programs and C programs are essentially the same. You could just as likely have assumed that Zig programs behave more like Rust programs (or Go programs), and at the time of deallocation there is just one pointer to the object, and voila, zero false positives.
I am not saying that a Zig program should behave like a Rust/Go/Java program, just that your arguments are begging the question. You start with the assumption that Zig and C are essentially the same and then use the resulting conclusions to shore up that very assumption. But Zig programs are about as different from C as they are from Rust.
You insist that if you're not Java or Rust then you must be C. Zig is so revolutionary precisely because it works like none of those. Now, I don't know if Zig's revolutionary design is revolutionarily good. It's far too early to tell. But basing your criticism on a false premise completely misses the mark.
Of course, it's okay to be skeptical of a new idea, just as I'm skeptical that a type-level shrine for accidental complexity is the way to go.
> But the right conclusion to draw from this is that Zig should use a tracing garbage collector.
I disagree, but that's a whole other discussion. But since you seem to claim that such a thing could exist and effectively work for Zig and not for C, it seems you accept both of my points: 1. that pointers in both languages are fundamentally different, and 2. that this can be exploited for analyses that are completely different in their effectiveness than those feasible for C or C++.
That’s the case in production mode. Your link says that checks are added for overflow in debug mode. Presumably that also includes underflow.
This is why Zig maintains these checks in ReleaseSafe mode and gives you wrapping operators (+%=, etc) when that's what you want.
So you’re saying that I was correct. Good to know.
[profile.ReleaseSafe] overflow-checks = true
And of course Rust has both Wrapping types and wrapping versions of each integer operation if you want those, although I think we can be confident that the sort of person who is shipping release builds with overflow checking isn't thinking far enough ahead to have decided what should actually happen when their variabes under/overflow so giving them the option is redundant.
Given that the problem here is they're Wrangling Untrusted File Formats and, unsurprisingly, they failed to do so Safely, they likely should have used WUFFS rather than Rust, or Zig, or any general purpose language.
The equivalent underflow mistake in Rust probably panics. However the RCON trick at the heart of Steam which is powering this lets you run any command as the victim if they accept your "invite" so crashing their game is almost certainly already an option without the underflow.
tl;dr nearly all of Rust is geared towards trying to solve problems I am not interested in, but it isn’t really the ‘borrow checker’ area that I personally find most off putting.
First, I am of the opinion that any centralized package management inevitably leads to heavy reliance on those packages. The comments of ‘there is a crate for that’ or ‘just look on NPM’ or ‘it’s on PIP’ are often the first answers anyone gives to people looking to write code to do a certain thing. That leads to towers of imports and dependencies. I am fundamentally opposed to not writing the simplest and most specific code for any given problem. Package management is a type of institutional environment directly opposed to that fundamental opinion. I am not interested in generic, allegedly re-usable, solutions, I want the most simplistic/direct solution/implementation I can code. Now sometimes that solution is a library and sometimes that library is complex, but that is a decision that is made purposefully and with the knowledge that the dependency MUST be managed as though the code was written by you. I think the package management paradigm being used/targeted by most modern languages assume that by using a dependency means I am outsourcing a potion of my code to others and that I am willing and able to change my code according to what changes with the dependency. I tend to only use direct source dependencies that I can make completely local and that code just becomes part of my program, and yes that means there are no updates to that code I do not make myself. An update of the dependency itself will only be included after reviewing the update as though it were a new dependency, i.e. purposefully and by me, not whenever the vendor wants and not whatever the vendor chooses.
I am aware that there is no requirement to use package management to incur what I find are negatives, but I truly believe that the environment and common usage patterns push development practice towards the type of coding practices I dislike.
Second, for work purposes, I can not use any code I can not warranty as fit for purpose and conforming to terms of a given contract. This means that any code which has a ‘no warranty’ statement in the license (which is almost all library code in every language regardless of distribution mechanism) I can only use once I have reviewed enough of it and am confident enough in it that I can put specific functional guarantees in place regarding specific performance of contract terms. Package management systems, and their accompanying practices, are not particularly appealing when the package management strategy seeks to continually push updates and continually seeks to stack reliance on large dependency trees. I am often better off designing and programming something myself that attempting to review to an acceptable degree a tangled mess of dependency.
I find that, while often an explicit non-goal for most programming languages, I would best be served by explicitly bare source code libraries, which are stored and compiled locally, and are incorporated into a project purposefully and in the same manner as files authored by me. Again, I am aware my requirements are not industry standard and my opinions are not mainstream, but there ya go.
This in fact _hurts_ maintainability as you need to manage this heavy, recursive dependency tree. Ultimately it's not hackable. You can't just change that function to print a warning, rather you have to edit an entire library and work through it's taxonomy.
I'd note that Python is much better than others you list at this. Idiomatic python tends to be "batteries included" and shouldn't use too many dependencies other than those that are crucial to the project (ex Numpy). It's probably also because Python's package management story isn't as polished as others.
You want the rural living version of programming languages. Package management is a dense urban core.
1. Don't assume that packages come from Github. Or from any specific URL, for that matter. Or that they're managed in Git. Make it controllable with a config file, and ship a default config file. Let a project-specific config file override a site-wide one.
2. Make sure packages can come from multiple sources.
3. Make it possible to use packages from the local filesystem. Support relative paths.
4. Don't try to do network access unless I specifically ask for a fetch/download/update/etc. I want to be able to extract a few source tarballs somewhere, and then run a build that doesn't require me to be online.
5. Have a good story for working with non-Zig libraries. Like if I want to make a FUSE filesystem or something, it probably depends on having libfuse and libfuse-dev installed from my distro package manager. Zig's package manager doesn't need to interact with apt-get or whatever, but it would be great if it knew how to find local headers and libraries.
6. Include a way for me to generate a source-tarball that bundles the sources for all of my program's dependencies, so that I can archive it and reproduce a build later even if the upstream sources go away.
7. Include a license tracking/reporting mechanism, so that I know the licenses of all software I've pulled in as a dependency (whether directly or indirectly). I don't want a situation where I have some dependency that has another dependency on another dependency, and 3 levels down there's something that's got a license I can't ship. The Yocto project did this in a very smart way, and it's a good model to copy.
8. Have a way to express build-time dependencies and run-time dependencies. Those aren't necessarily the same thing, and might not even target the same platform.
I'm sure you mean negative externalities that your customers aren't equipped to appropriately price.
Ever heard of Purify and MemCheck?
Most people keep ignoring such tooling to this day, if it isn't enforced by the language, or CI/CD pipeline, it just gets ignored.
Just like the strings/arrays, naturally one can make use of something like SDS in spirit, most don't.
As for the rest, I really don't get the point, given the plethora of such tooling for C and C++, both as free beer and commercial for the last 25 years.
The right place to enforce these things are in the language and compiler (which I think you agree with). Make it impossible to make mistakes. Which is also why I'm not that excited about Zig. But the point is the tooling story for C isn't a land of puppies and roses, even in 2021.
Right now, in regards to security in systems programming, Zig is no better than using Modula-2 compiler.
Also there a list a FLOSS static analyzers: https://en.wikipedia.org/wiki/List_of_tools_for_static_code_...
I use sanitizers, but they're dynamic as well. There's also additional work to make them work with custom allocators, but that's not too surprising. I am excited for the day I can finally use the ARMv8.3 (or is it v9?) feature that adds essentially HW acceleration so the overhead becomes ~1.2x instead of 5x.
If it were possible to solve use-after-free this way, Microsoft/Apple/Google would have done it long ago for C++.
But more seriously, I guess the general-purpose-allocator is mainly useful for catching memory corruption issues during development, despite the name.
If you follow the strategy behind Zig's "general purpose allocator" to its logical conclusion, you get Google's HWASAN for C++ [1]. It is much more advanced than an allocator that simply quarantines memory forever and uses probabilistic checking to mitigate the problems of quarantining, trading soundness for performance.
[1]: https://clang.llvm.org/docs/HardwareAssistedAddressSanitizer...
Edit: I might misunderstand your comment, ignore me if so.