Zig's multi-sequence for loops
kristoff.it
kristoff.it
for (const auto& [x, y] = std::views::zip(a, b)) {
/* ... */
}
If it is required for optimization to not check for reaching the end of one of the ranges then it can be achieved with something like:
for (const auto& [x, y] = std::views::zip(a, std::ranges::subrange(std::ranges::begin(b), std::unreachable_sentinel))) {
/*...*/
}
(loop
for x in a ; on each iteration, steps x to next element of a
for y in b ; same thing
do ...)
I think lisp’s loop isn’t hard-coded in the language, but defined in the standard library. See for example the loop implementation at https://github.com/sbcl/sbcl/blob/master/src/code/loop.lisp (about 2,000 lines because loop is a monster/very flexible (pick whatever you prefer. I would pick both))
https://www.kylheku.com/cgit/cppawk/tree/share/cppawk/includ...
The bulk of the file is defining the clauses, each of which requires six lines: six simple macros have to be defined. (Kind of reminiscent of the five SETF expanders in Common Lisp for defining a new place.)
This is thoroughly documented by a man page, including user-defined clauses. An complete example is given how to make a cause which iterates over alphanumeric strings, as in:
BEGIN {
loop (alpha_range (x, "aa0", "cc9"))
print x
}
;; the exprs evaluate to lists, and the vars step over them
(mapc-loop (expr1 expr2 ...) (var1 var2 ...) body)
(mapc (lambda (var1 var2 ...) body) expr1 expr2 ...)
(defmacro mapc-loop (exprs vars &rest body)
`(mapc (lambda ,vars ,@body) ,@exprs))
Seems hard to make it any shorter. Well, I guess you could remove "const" if you want.
for (const auto &[x, y]: zip(a, subrange(begin(b), unreachable_sentinel))) {
/* Do something with x and y */
}
So much C++ code can become readable by adding `using namespace std::something`.
Other languages like Java have imports scoped a to a single file, so this is not a problem.
It makes sense to use std::vector in a .h(pp) file, but in the .cpp you should be able to `using namespace std`, right?
Not if a lot of your code is in the form of templates. You have to put those in headers.
Some things take time when trying to avoid Python 2 / 3 scenarios.
Using in a cpp file => absolutely fine.
definitely not as soon as you want to do unity/jumbo builds (which are in my experience the n°1 thing to do to get fast CI builds)
Java avoids this somewhat because functions are always in a class, so there is a little bit of extra namespacing, even if you've imported the class.
import someclass.doThing;
Or maybe that's what you meant by somewhat :)
In a header it's generally advised to not add using namespaces.
However, I appreciate the grandparent comment including the namespaces in the example, so I can see exactly where things are coming from.
https://dlang.org/phobos/std_range.html#zip
Sorting two arrays in parallel:
import std.algorithm.sorting : sort;
int[] a = [ 1, 2, 3 ];
string[] b = [ "a", "c", "b" ];
zip(a, b).sort!((t1, t2) => t1[0] > t2[0]);
writeln(a); // [3, 2, 1]
// b is sorted according to a's sorting
writeln(b); // ["b", "c", "a"]Doing things like `a = @zip(some_list, some_other_list)` can be reasoned about in multiple ways, some of which involve silently calling malloc. It's particularly unclear what could be done with `a` afterwards. Zig hates that kind of ambiguity and is happy to err away from flexibility at times.
Although- thinking about it, that may rely on the borrow checker (move semantics specifically)
The impl of the default `next` is:
fn next(&mut self) -> Option<(A::Item, B::Item)> {
let x = self.a.next()?;
let y = self.b.next()?;
Some((x, y))
}
I don't actually know whether that applies to Zig though
It can't zip more than two per se, but you could zip the result of the first zip into a third and get ((item1, item2), item3). You could then map these if you wanted, to flatten them into a single tuple .map(|((item1, item2), item3)| (item1, item2, item3))
Of course there's a trade-off here between ergonomics and generality
FWIW that's more or less what `itertools::izip!` does for you, it just chains `zip`s then "splats" them using a `map`.
Right; my question is, suppose you're iterating over two slice iterators - won't each call to `a.next()` and `b.next()` have to check whether that sub-iterator is done? One of the benefits of the Zig approach is that you can iterate over an arbitrary number of slices and do one check before entering the loop, followed by the compiler emitting unchecked index access in the loop. So it basically compiles down to the equivalent of a C `for` loop.
`zip` yields exactly the same assembly as a loop over the index range with an unsafe item access: https://godbolt.org/z/7ebfxbhxc
You trade some special-case syntax and ergonomics for that generality, but it is very general even if not all of it is optimized in the same way
[0] https://github.com/rust-lang/rust/blob/f540a25745e03cfe9eac7...
The itertools crate has a multizip function/macro that allows zipping as many iterators as you'd like without nesting zips inside of zips!
I really recommend checking out itertools if you use rust and like iterators! It's a direct dependency of rustc itself too, which suggests its a trusted project (it may even be maintained by rust devs, not totally sure).
That's what I mean by straightforward. You'd have to argue about the memory layout for an iterator, that you call functions on it etc. Lots of small decisions. In zig those things would usually be part of the standard library, not language features.
If you want to call a next() function on a thing in zig, you want to be explicit. That's what the language means by "no hidden control flow". You'd use a while loop if you wanted to iterate on a hashmap for example.
In some sense, this is less elegant, but it's usually more obvious what the compiler would end up doing in any bit of code.
Does it? Rust seems happy to allocate silently all the time.
let x = String::new(“hi”);
let y = vec![];
Zig’s requirement of explicitly passing around an Allocator type removes any ambiguity completely.
But closures don't require allocation, iterators don't require allocation, async doesn't require allocation. Copy semantics also don't allow allocation- implicit copies can only happen for data structures that are bitwise-copyable, which is enforced by the compiler. For copy-with-allocation you have to implement the Clone trait, and then invoke it explicitly with the .clone() method
But the original context was a question of philosophy, so I was only speaking to Rust's overall philosophy
[0] Technically I think if you're using no_std you won't have access to any standard constructs that allocate (which obviously will prevent their use at compile-time), though I believe you're still allowed to eg. call out to foreign functions manually that would allocate. And of course, this still isn't as granular as Zig's allocation-control.
Indeed, because you only have core, and not alloc, which is where the allocator lives.
Mostly, core looks very similar to the functionality you can get of the same types in std - except that of course types which allocate are missing (Vec, HashMap, Box, String, etc.) and all the types which reflect operating system services are likewise missing (UdpSocket, File, Mutex, etc.)
However there are deviations. For example, Rust's slices in core don't have a sort() method. Why not? Rust provides a stable sort which uses an allocator because that's much faster, it doesn't bother providing a not-so-good stable sort that can work without using an allocator, if you can't afford an allocator but you want stable sort that's a rather niche case, Rust won't solve it. Rust's sort_unstable() doesn't need an allocator and so that is provided on slices even with only the core library.
Rust For Linux, the Linux kernel's Rust, has core, plus its own somewhat customised twist on alloc, plus an entirely custom kernel library. In Rust For Linux, the allocators are all explicit because Linus hates implied allocation. So for example in Rust For Linux a Vec doesn't have a push() method, because push() on a Vec can cause the Vec to grow, which is an allocation - instead Rust For Linux provides Vec::try_push() which fails if you'd need to grow the Vec before it could successfully push.
vec![] says I want a Vec with nothing in it. That doesn't require an allocation. In effect you are asking for three things, a pointer, of the correct type but not actually pointing at anything, and two integers (length and capacity) which are both zero. These three things don't live on the heap, they're a local variable on your stack.
Rust's first question here is: A vec of what? If we wrote something in those square brackets, it could guess the type of that, but we didn't. It may be able to infer the type from what is pushed into this Vec named y later, except we didn't make it mutable so we can't push anything into it - or if it's returned, from the return type (Rust does not allow function parameter or return types to be inferred). If Rust isn't able to infer the type, that's an error during compilation.
let mut y: Vec<()> = vec![];
let mut y: Vec<u8> = vec![];
let y: Vec<u8> = Vec::with_capacity(1);
Not really...
var list = std.ArrayList(u21).init(std.testing.allocator);
try list.append('a');
In both languages you still need to know the specifics of the data structures you are working with, the only real problem that Zig solves is to actually allow you to use custom allocators with the std collections, which is something that is still unstable in Rust.
That is, unless you explicitly handle OOM conditions inside your construct, e.g. 'crash if you're OOM', which isn't typical in zig code. All code I interact with will return an allocator error if allocation fails.
Whether a clearly visible `vec.push(x)` call is silent to you depends on your viewpoint. I'd say it's recognizeable as potential allocation, it has to be when pushing to a dynamic structure of arbitrary size - but I also understand how this can be seen differently. This is, I think, where Rust and Zig differ. Rust makes most expensive operations explicit but still allows things like this or macros, Zig tries to avoid even that.
one = (1..3)
seven = (7..10)
(one.to_a + seven.to_a).each {|n| puts n}
Yup, it works. First time I ever used refinements.
module JoinRanges
refine Range do
def +(other)
self.to_a + other.to_a
end
end
end
using JoinRanges
one = (1..3)
seven = (7..10)
(one + seven).each {|n| puts n}I watch language after language add sugar to maintain the appeal of their product, one niche group or application at a time. It turns into a death by a thousand cuts, or by a thousand sugar cubes. Most languages start out simple and appealing and understandable, an increasingly short amount of time later, they've layered on "helper" after "helper" to the point it takes a bit of expertese to consume the language effectively.
I dream of a world where we'd measure languages by the complexity of their ASTs rather than their popularity on a TIOBE or StackOverflow index.
To me, the followin is a bit of syntactic sugar that I think is the kind of transcendental "go big/basic with it" that I hint at.
Some time ago, I worked in a language that had this idea that any composable block of code could be captured as 0-N statements between the characters [ and ]. They thought they were being clever and called it a "block of code". Which I thought was cool, because it looked like a block. Pedants called it a BlockClosure. If you wanted to pass parameters to one of these, they used a colon denoted list. So a two arg block might look like
[:a :b | <code goes here> ]
So yay, pass a closure to a service, and it "captures" the values be invoking said closure with arguments.
And then the authors thought, okay, enough sugar for a few days, let's just use this. I mean really really really use this.
You can use a two arg block like that for a zip function of course, but why limit it to iteration? Use it in the standard library to implement the "for each" function. Which when you looked at was just that "how dare they not have a for syntax" while implementation. But because it wasn't embedded in the syntax, you could copy/paste/modify to come up with a filter iteration. Or a reduce. Or a map. Or all kinds of interesting compositions "selectAndCollectAndReject" with 3 closures.
And why stop there? They decided, "let's just do boolean logic with these block things too". So where as most languages has special syntax for conditionals (and once they start, they're in competition with their peers to keep adding more and more of them (do while, case, if, if with N elses, on and on). But they just wrote it like
<condition> ifTrue: [trueBlock] ifFalse: [falseBlock]
Sure they optimized it, but from a linguistic point of view, it was the same thing as above. No new sugar was needed.
Whereas many languages have added sugar for optionals (usually involving ?s), this language, 20 years ago, was doing it with closures already. Someone noticed they could implement the following family of "functions"
ifNil: [nilBlock]
ifNil: [nilBlock] notNil: [:notNilValue | notNilBlock]
ifNotNil: [:notNilValue | notNilBlock]
Sure, not as terse as ? (which some endeavoured to deal with), but the language semantics didn't have to change each time there was a new thing to do.
I'm sure there's a Lisper out there that can write their analog to the above. Because it too, is was one of these "do much with little" langauges.
As mentioned in the article, data oriented design runs into the pattern of wanting to iterate over parallel arrays of data frequently.
Loris' blog post points out that the new for loops address the latter:
> In the multi-sequence for loop version it’s only necessary to test once at the beginning of the loop that the two arrays have equal size, instead of having 2 assertions run every loop iteration. The multi-sequence for loop syntax helps convey intention more clearly to the compiler, which in turn lets it generate more efficient code.
It also builds on existing properties of slices/arrays, rather than adding a new "enumerate primitive".
You get the same exact asm as the manual loop.
Of course, the idiom recognition seems to kick in, in both cases, as there's no actual loop here. I tossed in a +sum, which makes that fail, so you get some loops, check it out:
https://godbolt.org/z/1ddf5ded7
They are one instruction different in length, which is kind of amusing to me. Some small differences.
Sometimes you will need to be explicit about what you want to happen, and then a proper loop facility will let you do that.
for x in elms {}
for x in elms, i in 0.. {}
for (elms, 0..) |x, i| {}
^^^ ^
Ok it is in second position
fn func(x: i8, i: i8) void {}
for (elms, 0..) func
while (condition) : (i += 1) { if (foo()) |result| {
while (it.next()) |elem| {
bar() catch |err| …At the risk of sounding like a total ass: when I was checking Zig out (albeit briefly), it reminded me of the "Awful Taste, Great Execution" meme. I remember loving the ideas, but groaning internally when I saw how they need to be used. If someone is interested I could skim docs to see if I can still find an example of what put me off.
Things also seemed arbitrary and inconsistent sometimes, but that could've been a wrong impression since I didn't give it a proper shot overall.
It's a very new language, so I'm sure a lot of it is still in flux anyways. Also, this was all about a year and a half ago, maybe a bit more. So, maybe things improved since then.
The capture syntax is syntactically consistent across different constructs that support it, but it isn't semantically consistent. This consistency actually hurts because it's not expressing a general principle across the constructs, rather it's a convenience particular to each construct that we have unified syntax for, giving you the wrong impression the same sort of thing might be happening when it isn't.
The for capture takes elements of whatever kind in an array.
The catch capture only takes errors.
The if capture only takes optionals.
So, it's the same syntax for things that are totally semantically distinct. I get they're trying to make the language "guessable" to an extent and easy to remember/simple enough to fit in one's head, but imo this actually makes it harder to remember. I wind up getting confused because I have to stop to think about what gets captured and what it means etc. It's a writer convenience over a reader convenience. I'd much rather have to remember more, visually distinct syntax for different semantics than I would have to remember special, subtly different overloads of the same syntax across a range of semantics--ironing some syntax into the brain isn't as costly, imo as having to constantly remind oneself how these similar structures differ ever so slightly each time and untangle choices that are pragmatic but not semantically grounded.
All that said, the rest of the language is shaping up pretty fantastically.
Edit: As an example as to how it could be done differently, for null checks go just has the obvious "if (x == nil)". People complain about go's error check verbosity, but imo this is way clearer than zig's approach which requires remembering special magic "unwrap and bind to this only if not null" while saving only a few characters at most.
Also so many special characters for such a simple construct! While I do like {} languages, this has () || {} whereas e.g. Rust would only have {} in that case:
for x in xs {}
for (x, y) in xs.iter().zip(ys.iter()) {}
use std::iter::zip;
for (x, y) in zip(xs, ys) {}
I'll admit though the syntax is a little off putting. But that is a minor complaint. I know it's not 1.0 and there are still lots to do, but I'm curious if they do more for memory safety. With companies trying to avoid starting new code in memory unsafe languages if they don't have to , I wonder if that will hurt Zig adoption. Right now it just seems like their approach is, reduce Undefined Behavior as much as possible, make the "correct" way of programming the easiest, and have array bounds on by default. Will this be enough to make the language "memory safe" enough?
I'm not convinced that this needs to be in the type system.
Nonetheless it's not there yet.
Can you sketch out why hooks are necessary on the proposals, like 14656 ? So far I have not read a justification, what the advantage of coupling it to the compiler provides as performance gains on concrete examples. Afaik, Rust has lifetime checks decoupled to parallelize compilation and I have not seen research or a list of the possible optimizations with synthetic benchmark results.
> I'm not convinced that this needs to be in the type system.
If you want to prevent static analysis semantics becoming an incompatibilty mess like C due to 1. unknown and 2. untrackable tag semantics, then you have to provide a at least an AST to ensure annotations have known + unique semantics as I explain in https://github.com/ziglang/zig/issues/14656#issuecomment-143.... I would say that this is a primitive quasi-type check decoupled from main compilation and could be kept very flexible for research experiments by having an uri + multihash (so basically what Zig uses as package).
More concretely: Stuff to look out for is RefinedC and related projects to have more strict C semantics + how to compose those things outside the regular Zig type system.
The nice thing is that Zig conventions make things just work, because you could choose the command for your code base and third-party code would not need to care about your chosen build commands.
The "binary lsp" will give the necessary info of what has changed etc, so the info is (eventually) incremental with minimal overhead (only ipc + optional network latency).
It seems like statically checked zig has a chance to be strictly easier to implement relative to statically checked C, and that's I think something to shoot for, especially since it could require very little on the part of the zig developers proper.
[0] - https://odin-lang.org
- The happy path in Zig encourages memory-safe programming and has some tools (runtime checks by default, "Release-Safe" builds with those same checks) for validating reasonable behaviors.
- The (explicitly cultivated) culture around constructs like Allocators helps one be very aware of the memory they're using and what they need to free.
- Conventions of the form `try Obj.init()` and `obj.deinit()` act a lot like RAII and further emphasize the happy path in ziggish code.
- Directly contrasting the rust I write for $WORK and the zig I write for me, especially when the actual memory layout matters (e.g., pick your favorite performance metric) I don't have a substantial difference in the memory safety bugs I write in either language. Rust is actively trying to improve that story and reduce the need for unsafe blocks for atypical constraints, but IMO it's not good enough yet.
- Zig has some shocking footguns that combined with the current pre-1.0 state of code and documentation can introduce issues you probably would not expect. E.g., most meaningful uses of async frames require them to be stored somewhere, but applying the same coding techniques you would in most Zig code and adapting the docs to do so you'll write something that creates a frame and puts it somewhere. Doing so will actually work for small examples, but as soon as that frame does something with a pointer to anything in its stack it'll blow up in production because it's retaining a pointer to the old memory you copied from rather than the new place you copied it to. There are easy solutions, but without in-depth knowledge about exactly what's happening (which is hard to obtain from the current docs) it's hard to realize you would have to find a solution (admittedly, the Zig Discord is __amazing__ at quickly helping people with such issues, so it's not necessarily a huge deal since it's solvable that way now and probably solvable with better docs post-1.0).
- I've only said anything about single-threaded memory safety. Rusts compile-time reader-writer locks are next-level when working on nitty gritty concurrent code. I've hacked together some interesting parallel Zig projects, and I'm like 99.9% sure they're correct now, but it was an adventure to even get it that far, and I'm still not 100%. I haven't been actively following Zig news for a couple months, but last I heard they're planning something important with async/await to improve this story, but it's not there yet.
I'm of the opinion that such companies or persons that would use it, are not as concerned with memory safety. Probably their use cases are more specific, limited, or narrower. Else they would likely go with more convenient GC using languages. Of which, various such languages are also compiled with optional GCs that can be turned off (Dlang, Nim, Vlang). That, or deal with the greater complexities of adopting Rust or Ada, for primarily attempting to gain such greater memory safety. Per that part of your comment, do think the adoption will be smaller, more niche, and/or we will see it hit a lower ceiling.
Yes, a flexible solution (a zip function) would probably need iterators but why would introducing them be such a big problem? (FWIW, I know that one can emulate looping over iterators with while() and optionals[0] but it feels a bit dirty.)
More generally, my biggest gripe with Zig has been a lack of expressiveness: Things like deep equality checks between structs, arrays and optionals; looping over different kinds of containers; … should just work™. Sure, I understand that Zig doesn't want hidden control flow but OTOH explicit control flow everywhere often just gets in the way of readability and of implementing business logic. I usually follow the approach "Implement first, optimize later" but with Zig the implementation will look completely different depending on which optimization or data structure I choose, so I need to think about optimizations from the start if I don't want to rewrite everything later. I should mention, though, I'm very used to Python these days and haven't written C or C++ in ages, so maybe that sort of culture shock is somewhat expected.
Anyway, I'm still excited about the language and my impression is that Andrew Kelley is very open to new suggestions and new ideas, so things will certainly still change in one way or another till v1.0.
I can tell you right now that while there will be many upcoming language changes, none of them will be comfy to Python programmers. Zig is very much an imperative language. Or perhaps think of it as a declarative DSL for outputting machine code.
and thanks for the laugh
Just to be clear, I didn't mean any offense and maybe my critique wasn't as well-balanced as it could have been. So far, coding in Zig has been a fun ride, despite the occasional obstacles I've run into!
> There is no "just work" for deep equality
Say I have two variables A and B of the same struct type. Each points to a finite region in memory of the same size. Why can't I just compare these two regions using `A == B` to make sure they are equal? Yes, one can obviously come up with alternative definitions of what equality might mean for structs (only compare certain subfields etc.) but wouldn't the aforementioned definition be a good default that would work in the majority of cases?
Alternatively, there is `std.testing.expectEqualDeep()` which walks through all fields but as far as I know there is no equivalent for production code(?)
> none of them will be comfy to Python programmers
Oh I think the multi-sequence for loops feature already makes Zig more comfy! :)
Just to be clear: I wouldn't want Zig to be another Python. While I like Python from a developer experience POV, it is dictionaries and magic methods all the way down and often unnecessarily slow and complex.
I still think one could find a good balance between DX and full low-level control, though. One could e.g. have one convenient way to express a certain problem that gives you medium control over performance (e.g. the `==` example above) and one or more fine-tuned, but possibly less concise ways of expressing the problem that provide full control but require more lines of code. In the struct equality example the latter would mean defining some kind of `eql()` function that would be optimized to the struct type (e.g. compare certain fields first as they are more likely to differ etc.). Would this violate the Zen of Zig?[0]
> Only one obvious way to do things.
After all, there is also
> Favor reading code over writing code
Right now, at least, I often run into situations where I don't know of any obvious way to solve my problem. Then I end up writing lengthy code to tell Zig what I want and end up with code that's so-so on the fun-to-read scale.
Why is shallow equality useful?
You could have `A == B` be true, but then as soon as you wrap pointers to them in C and D, now `C == D` is false.
> Why is shallow equality useful?
Of course deep equality checks are useful, too, I'm not denying that. But in many cases you are not dealing with several layers of structs nested through pointers. You just want to compare simple structs and be done with it, without having to write an eql() function for every single case.
Besides, many other operations in Zig operate directly on the memory, so why should == be any different?
My apologies for the confusion!
Deep equality checks don't prevent the need for minor rewrites in other cases, like when you change a widely used type's definition.
Rewrite code during optimizations or rewrite the project once Python isn't fast enough? I don't know, both can be very cumbersome.
I am confident this is a mistake. Every time you make a new kind of "thing" in your language somebody will want to do all the same stuff with it that they did with the other things, such as integers, ie in this case store a range in a variable. Ideally you'd just always be able to do that, see Lisp, but it can get very unwieldy, thus this is a reason to avoid making new kinds of thing so the issue doesn't arise.
C++ chooses to actually do the heavy lifting here, which is why std::format (and its inspiration fmt::format) was such an enormous undertaking -- C++ can express the idea of a function which takes a variable number of arguments and yet all those arguments are independently type checked at compile time, not via compiler magic but just as a normal feature of the language. This is an enormous labour, and because they don't have any way to fix syntax issues the resulting problem accumulate forever in their language so I cannot recommend it as a course to other languages. It's like the Pyramids, do not build giant stone tombs for your leaders, this is a bad idea and your society should not copy it - however, the ancient Egyptians already did build giant stone tombs and they're pretty awesome to look at.
Anyway, Rust chose to make its half-open range type std::ops::Range an actual type which you can store in a variable, pass to functions, modify etc. as well as using it in a for loop. Obviously don't copy Rust here exactly, for one thing Range should probably be IntoIterator, not an Iterator itself if they had it over, but you will wish this was an ordinary type in your language, so, just do it now.
let a = 0..4; // The Range starting at zero and (non-inclusively) ending at four.Though in fairness getting an iterator abstraction to the same efficiency as a for loop requires pretty brutal optimisations, frankensteining your for loop, a lot less so.
Then you could pass them around, but only at comptime, which will achieve many of the things you expect.
There's also nothing stopping you from creating an iterator interface in userland.
I don't see any way you could do that with these ranges, and I'm unsure to what extent you can do this for any of the built-in types without their co-operation. If it needs co-operation then there's actually a lot stopping you from creating a useful iterator interface.
Also there are a bunch of plausible iterator APIs ranging from a very narrow API like Rust's Iterator trait to something as broad as C++ "iterators" which are effectively just (sometimes magic) pointers.
How about Nim's inline iterators?
That's the role of the compiler and if Rust or C++ are anything to go by, it is more than capable to "see through" iterators.
for(@as([10]void, undefined)) |_, idx| {
_ = idx;
}
The implementation of this proposal brings with it the additional benefit of making index capturing less mysterious.
> (i.e. you will get a panic in safe release modes)
Should I take that to mean there is an unsafe release mode without the bounds check? But UB is mentioned too. Is there UB!?
It's 2023; I think we can afford a single branch to avoid UB, even in release mode.
I think that disabling safety checks is a thing you should only do if you are studying the cost of safety checks (i.e. a compiler switch only available to compiler engineers). IMHO, the whole point of safety checks is to find the bugs that are in your program[1]. Crashing it safely with an exact source stack trace is the nice way of both motivating you to fix it and also helping you fix it.
[1] And there are bugs in your program. Right now. Bugs. In. Your. Program. Running without safety checks is like closing your eyes and rolling the dice.
[1] https://ziglang.org/documentation/0.10.1/#setRuntimeSafety
Arguing that safety checks should always be enabled doesn't really make sense. Context matters.
Yes, it's called ReleaseFast.
> It's 2023; I think we can afford a single branch to avoid UB, even in release mode.
Zig has 3 release modes: ReleaseSafe, ReleaseFast, ReleaseSmall. If you want the safety checks, just use the first.
It might even be $currentYear, but many of the latest AAA games still don't always run at more than 60fps on my fairly powerful machine and I sincerely hope they were built with all the optimizations enabled.
Sure, but I don't agree that disabling safety checks is an "optimization". It is a regression in functionality that is betting on nothing going wrong.
Bounds checks do not cost much[1]. Maybe if a bounds check disables vectorization[2].
[1] https://blog.readyset.io/bounds-checks/
[2] https://github.com/matklad/bounds-check-cost
There are a lot of techniques to remove bounds checks, e.g. in counted loops [3][4].
[3] https://ieeexplore.ieee.org/document/5381765
[4] https://en.wikipedia.org/wiki/Bounds-checking_elimination
This can even be decided on a scope-by-scope basis if so desired.
The same is true of poorly performing programs. My computer's resources are not the programmers' to waste, yet they routinely do waste it to save themselves time[0].
> I don't know how as a profession we're so cavalier with shipping exposed whirling knives, but we are.
That's a separate problem than not handcuffing programmers and forcing them into safety checks. Why should Zig force this?
Like, I just don't even get what you're complaining about here. The default build mode and the recommended release one insert the check. Checks can additionally be enabled and disabled on a scope-by-scope basis. What exactly do you want? Just eliminate ReleaseFast as an option and give people more reasons to go back to footgun-laden C because it'll be the only way to eliminate a bounds check in a tight loop hot spot?
[0] Yes, I know this isn't due to safety checks in the vast majority of circumstances, that's not the point. I have nothing against safety checks, my problem is with the mentality that it should not be possible to disable them. Even Rust has `unsafe`.
> Rust evangelists need to be careful because in their zeal they have started to cause subtle errors in the general knowledge of how computers work in young people's minds. Ironically it's a form of memory corruption.
I'm not even a zig user or fan or anything and I don't have any real opinion about Rust, either, except for completely agreeing with this analysis based on how I've seen Rust evangelists talk online. I'm not sure what the solution to this is, but it seems like it's just going to get worse over time as Rust becomes more popular and gains market share.
What alternative name would you prefer to express the collection of memory safety features in programming languages?
In most engineering professions, it's the engineer's responsibility to ensure appropriate levels of safety, not the CAD software used to build the blueprints. But every situation doesn't have the same level of safety required; backyard sheds don't have the same needs as skyscrapers.
I actually do think it is the responsibility of the language and runtime system to ensure some base-level safety of programs. The one constant over the years is that programmers keep making mistakes. No matter how much they keep yelling "trust us", they (we) just keep screwing up. That's not to pillory us programmers. It's just the facts that everyone screws up. In some sense, engineering is putting processes and procedures and checks in place that move human fallibility out of the critical load-bearing situations so that a simple whoops or memory slip doesn't kill people or ruin things.
With bounds checks: game crashes, meaningless error message given to the user.
What am I missing?
Sometimes without bounds checking you get an exploit instead of a crash.
Without bounds checks: I join a multiplayer lobby, and the next thing I know my computer is part of a botnet.
This isn't an imaginary fear, it has happened many times. Some examples from a brief search: https://gridinsoft.com/blogs/rce-vulnerability-in-gta-online... https://www.polygon.com/22898895/dark-souls-pvp-exploit-mult... https://security.gerhardt.link/RCE-in-Factorio/
I am not claiming all of these would've been prevented by bounds checking arrays, or even memory safety in general. The point is that security is not optional just because it's a game.
I'm not suggesting that shipping without bounds checks is wise or leads to a better product. However I do think with /some games/ security is basically not a concern.
I think it's more like (assuming it does actually go out of bounds at some point):
30% chance of core dump right away
20% chance of core dump at some point after errant write
40% chance it never crashes in testing
5% chance it doesn't crash the first year after shipping
5% chance it never crashes
With an explicit bounds check, all of these scenarios result in a crash at the exact location where the program first violated safety[1]. The developer gets a source-level crash and doesn't spend the first 20 minutes just trying to figure out what the crash dump even means.
[1] Hopefully with a complete stacktrace, maybe even the index and length values!
It's time we recognized that all our tooling should be designed to help us programmers who do have bugs in our program. Like, this crashing part is the normal part that all the tools should help deal with.
https://lispcookbook.github.io/cl-cookbook/iteration.html#lo...
Edit: 27 years since I was paid to write Lisp....
Most languages have a function called zip or something similar (https://hackage.haskell.org/package/base-4.17.0.0/docs/Prelu...) which handles pairing sequences, to be composed upstream of the iteration proper.
It is more limited than the one in common lisp, since it turns all loops into tail recursion and this uses no mutation outside the higher-order looping constructs. It is written mostly using the hygienic syntax-rules facility (yes, it actually works) and I haven't managed to get it to produce slow code yet.
It's a distributed database for tracking accounts and transfers of amounts of "thing"s between accounts (currency is one example of a "thing"). You might also be interested in our FAQ on why someone would want this [1].
[0] https://github.com/tigerbeetledb/tigerbeetle#quickstart
[1] https://docs.tigerbeetle.com/FAQ/#why-would-i-want-a-dedicat...
And you can model external accounts that have their own confirmation process using our two-phase transfer support.
Yes, exactly. You can think of TigerBeetle as your internal ledger database, where perhaps in the past you might have had to DIY your own ledger with 10 KLOC around SQL.
And to add to what Phil said, you can also use TigerBeetle to track transactions with other parties, since we validate all user data in the transaction—there are only a handful of fields when it comes to double-entry and two-phase transfers between entities running different tech stacks.
The TigerBeetle account/transfer format is meant to be simple to parse, and if you can find user data that would break our state machine, then it's a bug.
Happy to answer more questions!
- LMAX inspired
- Static memory allocation
- Zero copy with Direct I/O
- Zero syscalls with io_uring
- Zero deserialization
- Storage fault tolerance
- Viewstamped Replication consensus protocol
- Flexible Quorums
- Deterministic simulation like FoundationDB
First, because I'm a strong believer in defense-in-depth. Secondly because both disk corruption and network packet corruption happen. Alarmingly often, in fact, if you're operating at large scale.
For example, our deterministic simulation testing does storage fault corruption up to the theoretical limit of f according to our consensus protocol.
Details in our other reply to you.
"This means absolute trust in data read from disk or received from other nodes?"
For example, where most databases will have a network fault model, TigerBeetle also has a storage fault model (https://github.com/tigerbeetledb/tigerbeetle/blob/main/docs/...).
This means that we fully expect the disk to be what we call “near-Byzantine”, i.e. to cause bitrot, or to misdirect or silently ignore read/write I/O, or to simply have faulty hardware or firmware.
Where Jepsen will break most databases with network fault injection, we test TigerBeetle with high levels of storage faults on the read/write path, probably beyond what most systems, or write ahead log designs, or even consensus protocols such as RAFT (cf. “Protocol-Aware Recovery for Consensus-Based Storage” and its analysis of LogCabin), can handle.
For example, most implementations of RAFT and Paxos can fail badly if your disk loses a prepare, because then the stable storage guarantees, that the proofs for these protocols assume, is undermined. Instead, TigerBeetle runs Viewstamped Replication, along with UW-Madison's CTRL protocol (Corruption-Tolerant Replication) and we test our consensus protocol's correctness in the face of unreliable stable storage, using deterministic simulation testing (ala FoundationDB).
Finally, in terms of network fault model, we do end-to-end cryptographic checksumming, because we don't trust TCP checksums with their limited guarantees.
So this is all at the physical storage and network layers.
"Zero deserialization? That sounds rather scary."
* assume a non-Byzantine fault model (that consensus nodes are not malicious),
* run with runtime bounds-checking (and checked arithmetic!) enabled as a fail-safe, plus
* protocol-level checks to ignore invalid data, and
* we only work with fixed-size structs.
* have a simple data model (account and transfer structs),
* validate all fields for semantic errors so that we don't process bad data,
* for example, here's how we validate transfers between accounts: https://github.com/tigerbeetledb/tigerbeetle/blob/d2bd4a6fc240aefe046251382102b9b4f5384b05/src/state_machine.zig#L867-L952.
In our experience, zero-deserialization using fixed-size structs the way we do in TigerBeetle, is simpler than variable length formats, which can be more complicated (imagine a JSON codec), if not more scary.
Oh, nice one. Whenever I speak with people who work on "high reliability" code, they seldom even use fuzz-testing or chaos-testing, which is... well, unsatisfying.
Also, what do you mean by "storage fault"? Is this simulating/injecting silent data corruption or simulating/injecting an error code when writing the data to disk?
> validate all fields for semantic errors so that we don't process bad data,
Ahah, so no deserialization doesn't mean no validation. Gotcha!
> In our experience, zero-deserialization using fixed-size structs the way we do in TigerBeetle, is simpler than variable length formats, which can be more complicated (imagine a JSON codec), if not more scary.
That makes sense, thanks. And yeah, JSON has lots of warts.
Not sure what you mean by variable length. Are you speaking of JSON-style "I have no idea how much data I'll need to read before I can start parsing it" or entropy coding-style "look ma, I'm somehow encoding 17 bits on 3.68 bits"?
Exactly! We focus more on bitrot/misdirection in our simulation testing. We use Antithesis' simulation testing for the latter. We've also tried to design I/O syscall errors away where possible. For example, using O_DSYNC instead of fsync(), so that we can tie errors to I/Os.
> Ahah, so no deserialization doesn't mean no validation. Gotcha!
Well said—they're orthogonal.
> Not sure what you mean by variable length. Are you speaking of JSON-style "I have no idea how much data I'll need to read before I can start parsing it"
Yes, and also where this is internal to the data structure being read, e.g. both variable-length message bodies and variable-length fields.
There's also perhaps an interesting example of how variable-length message bodies can go wrong actually, that we give in the design decisions for our wire protocol, and why we have two checksums, one over the header, and another over the body (instead of one checksum over both!): https://github.com/tigerbeetledb/tigerbeetle/blob/main/docs/...
So, how's the experience of implementing this in Zig?
And we're always learning.
But Zig is the charm. TigerBeetle wouldn't be what it is without it. Comptime has been a gamechanger for us, and the shared philosophy around explicitness and memory efficiency has made everything easier. It's like working with the grain—the std lib is pleasant. I've learned so much also from the community.
My own personal experience has been that I think Andrew has made some truly stunning number of successively brilliant design decisions. I can't fault any. It's all the little things together—seeing this level of conceptual integrity in a language is such a joy.
The advantage of Rust would be a nice and readable stack trace to the crashing method, but a core dump would've included even more information for the person debugging the binary, so I think it ends up quite even.
A reminder for all that have forgotten: UB is the one that can email your local council and submit a request to bulldoze the house you’re in. It is not a free core dump.
It is 100% well-defined behavior to dereference these pointers. It always segfaults, which as Jarred mentioned is a lot like a panic.
Rust evangelists need to be careful because in their zeal they have started to cause subtle errors in the general knowledge of how computers work in young people's minds. Ironically it's a form of memory corruption.
Is that guaranteed by the language semantics, or could it possibly change at some point in the future? If it's the latter, then yes, it is very much Undefined Behavior, and not guaranteed to segfault before opening the door for potential exploits.
On Windows and Linux this is the first 4KiB so range 0x0000 up to 0x1000, unless large pages are on (then it's even more).
On macOS in x64 this is the entire 4GiB memory space, probably a method to help developers port their 32-bit software to x64. I don't know what the zero page size on ARM is.
If your microcontroller doesn't have this guarantee, you can't make use of this feature.
Remember this gem?
https://kristerw.blogspot.com/2017/09/why-undefined-behavior...
Once you trigger UB, all bets are off and your code could do anything. A segfault just means you spun the roulette wheel, bet it all on red, and got lucky your house wasn't bulldozed.
Zig also uses LLVM under the hood, right? So it's subject to these same semantics. An LLVM pointer value cannot legally contain arbitrary non-null non-pointer integers such as 0x2. That's a dead giveaway of UB. And I doubt the emitted Zig code safety-checks every pointer dereference for a value less than 0x1000 before performing the dereference.
0x2 is a perfectly valid pointer value, it just happens to never be a good virtual memory address on modern systems where virtual memory is setup by the usual OSs, hence the fact that you can rely on it segfaulting.
Zig UB is not C UB. There is an entire language built on top of it. Just because something behaves a certain way in C, doesn't mean the same thing is true in Zig. Zig is no longer a code generator for C, it has switched to a self hosted compiler a while back. In fact, the language is rapidly progressing to the point where LLVM is a mere optional dependency.
I don't know the semantics around LLVM pointers. I don't see why 0x2 would be invalid, there are plenty of platforms programmed in C(++) that have a flat memory model. It would be quite painful to have a microcontroller where you can't send data to the output pin because LLVM decided that 2 is invalid (but 0 isn't). I've never seen LLVM complain about invalid dereferencing, though, it always ends up doing what the compiler tells it to do as far as I can tell.
Zig pointers will definitely cause UB but most Zig code shouldn't need them. Slices are actually bound checked and should probably be preferred in most cases of pointer arithmetic. Simple pointers can't be increased or decremented so you need to manually go through @intToPtr if you want to do real pointer arithmetic, which is quite unusable.
I haven't used Zig much so I don't know how many Zig semantics are copies of C semantics and how many are translated by the Zig frontend. However, "this is a bad/undefined thing in C so it must be a bad/undefined thing in Zig" is simply not true.
LLVM has rules about what is legal and what is not legal. If you follow the rules, you get well-defined behavior. It's the same thing in C. You could compile a safe language to C, and as long as you follow the rules of avoiding UB in C, everything is groovy.
Likewise, this is how Zig and other languages such as Rust use LLVM. They play by the rules, and get rewarded by well-defined behavior.
I will return the courtesy, with regards to my interpretation:
> An integer constant other than zero or a pointer value returned from a function not defined within LLVM may be associated with address ranges allocated through mechanisms other than those provided by LLVM. Such ranges shall not overlap with any ranges of addresses allocated by mechanisms provided by LLVM. [2]
[1]: https://llvm.org/docs/LangRef.html
[2]: https://llvm.org/docs/LangRef.html#pointer-aliasing-rules
- Any memory access must be done through a pointer value associated with an address range of the memory access, otherwise the behavior is undefined.
- A null pointer in the default address-space is associated with no address.
A null pointer (0x0) is associated with no address, therefore it has no address range. So if you do attempt a memory access (dereference), the behavior is undefined. QED. A naive translation to assembly would indeed segfault on a modern OS, but LLVM's optimizations are free to assume that code path is unreachable and do anything else.
Once the program is in this state, a bug of some kind is unavoidable. I don't take issue with that - what I take issue with is your claim that this behavior is well-defined, because it definitely is not. It would be equally valid for a null dereference to corrupt your program state or wipe your hard disk.
In Rust for example, derefencing a raw pointer is unsafe - because that pointer could have a value of 0x2 - which would result in undefined behavior according to LLVM.
tbh I'm surprised any of this is even up for debate. If you google "is segfault undefined behavior" you'll get 100 results telling you yes, yes it is.
I'm not sure how I can connect the dots any more clearly. Like gggggp said, it's baffling to see the creator of a popular language sweep the nasal demons under the rug and pretend that certain undefined behavior is guaranteed.
Calling such segfaults "safe" or "well-defined" is setting your users up for disappointment and CVEs, because a "well-defined" result is axiomatically impossible in the presence of undefined behavior. It's subtle, and if we were talking about a Java competitor maybe I could forgive the mistake. But if you're writing a low-level language it's important to understand how this stuff works. Ironically, he spread misinformation in the very post where he accused Rust evangelists of the same.
This thread is long dead and continuing the discussion seems futile, so I'll just leave it at that.
*excluding something silly like `raise(SIGSEGV)`
Do the docs actually define exactly which mechanisms external to LLVM count as allocating address ranges and which do not? It's possible that calling mmap and passing PROT_NONE does not count, for example.
I don't believe PROT_NONE suffices. The address needs to be accessible, not merely mapped. If reading through a pointer, the address must be readable. If writing through a pointer, the address must be writeable. This is why writing to a string constant is undefined behavior, even though reading would be fine.
Another issue is alignment. If you read from a `*const i32` with unaligned pointer value 0x2, the optimizer is free to assume that code path is unreachable and, you guessed it, bulldoze your house. If you get a segfault from reading an `i32` from address 0x2, you've already hit UB and spun the roulette wheel.
In theory the emitted code could check pointers for alignment and validity (in whatever platform-specific way) before accessing them, and simulate a segfault if not. Such checks would serve as optimization barriers in LLVM, and prevent these instances of UB. Of course Zig's current ReleaseSafe doesn't do this, and I think it would be silly if it did. But that's the only way you could accurately call segfaults "well-defined".
Then I guess it could be a language guarantee if Zig only supports/targets those platforms. However, considering how low-level Zig is, I doubt that that is the case.
It isn't, in the general case. But JavaScript engines do some dark magic with pointer packing / NaN boxing as a performance optimization (most things in the VM are single words, passing around a single word is usually way cheaper than full unboxing), and I suspect bun in occasionally running into issues where it gets returned something from the JS engine which it thinks is a pointer but actually it's a packed, special value. This is a logic error, that turns into a weird memory issue at the abi boundary, not a memory safety issue.
Not on every architecture, not in LLVM (even if well-defined on the underlying architecture), and not in C (even if well-defined in the underlying compiler backend).
TL;DR: Zig injects checks and aborts the program at runtime unless you specify that you wish to ignore the problem. This can be done explicitly within the code or by compiling under a build mode that ignores checks (unless specified manually).
Programs compiled as Debug and ReleaseSafe will terminate at runtime if UB is triggered. Compiling for ReleaseSmall and ReleaseFast will cause traditional C-style UB. If you care about your program doing what it's supposed to do, you use ReleaseSafe. Doing Release[Fast|Small] will do something similar to -O3 in other languages, which will often change behaviour.
Note, however, that you can compile your code under "just allow UB and see what happens" mode but still benefit from checked UB by setting @setRuntimeSafety(true); this will introduce the assertions despite the unsafe build modes you may specify.
It's like introducing a C++ compiler flag* telling the compiler "ignore exceptions and just continue". You know you're in for a bad time the moment you specify it, but it makes your program blazingly fast because it greatly reduces the amount of code to generate/checks to execute.
The main advantage of checked UB is that well-tested code can make use of the unchecked nature of these features for speed without having length check code blocks that need to be wrapped in debug #ifdefs or similar. Assuming you don't run test builds with checks enabled (and why wouldn't you) you'd catch these problems in your build pipeline.
This is different from the normal way of working with C and friends, where UB remains in debug/-O1 builds but just acts a little differently. Some compilers will insert breakpoints, others will ignore the problem like in release mode, nobody knows what will happen and your compiler can't detect this problem for you.
* note that -fno-exceptions exists, but that aborts the program rather than let it continue.
I can enable lightweight assertions in libstdc++ and libc++ and it makes C++ safer, but not in any way "safe". There are some flags that can be enabled to trap on some language UB too, without bringing in the heavy weight sanitizers.
So, if I read this correctly, barring the simple cases that Zig can detect at compile-time, this means that whether it's a UB (in the C++ definition of the term) depends on the flags specified by the author of the library and the person who compiles the final binary.
That's definitely much better than C++ UB. Still a bit scary, though.
I thought it was defined as an OOB access in non-safe modes, or a panic in safe-modes.
I do wonder about performance, though, as multiple array derefences may not be captured well by the L1 cache like a well-rounded struct might.
An L1 cache line is often 64 bytes long, enough to fit one of the "monster" example structs but never two. Performance in real life scenarios may actually increase if these structs are padded with an additional 16 bytes so none of the structs are on a cache line boundary.
So… zip?
Python’s does that, and for most other langages you could use overloading, basic macros, or traits trickery to get there if you really wanted to support unreasonable widths (IME you almost never need more than 3, and combining two zip/2 works fine then).
Unlike Python, you can't pass a zip generator around. It's just a for-loop. While zig loops are expressions, they only return a single value.
Does it, actually? Does Zig's built-in pseudo-zip outperform Rust's? Or C++23's?
> It's just a for-loop.
Except it's not "just" a for loop, it's a weird special case for a for loop. And one which is actively dangerous too.
You could also argue that a for loop which can only iterate over a _single_ sequence is a special case of a multi-sequence for loop.
> And one which is actively dangerous too.
In safe builds (ReleaseSafe, Debug) it will cause a controlled panic if the sequences are not of the same size. Most likely it's a logical bug if you iterate over two sequences of different sizes. In ReleaseFast the compiler will make assumptions to improve performance. If it's very important for your code you can force a certain code block to always have runtime safety. Yes, there are trade-offs, but I don't feel it's _unreasonable_.
When a "for loop" in virtually every language is not multi-sequence, not really. People expect a "for loop" to be a certain kind of thing.
In implementation, Python's zip will return a generator that is iterated over using the iterator functionality, while Zig's .zip is compiled as a loop. Python's iteration may be turned into a loop, it may be interpreted, or it may be turned into some other kind of bytecode, who knows. The standard cpython implementation is much more complex, though: https://github.com/python/cpython/blob/main/Python/bltinmodu...
Concatenating zip()s is an unnecessarily complex solution, both in terms of syntax and in code generated. In Python this may not matter because it's a relatively slow programming language in general (the language often being "glue between fast C methods"), but in Zig this can easily become untennable.
I also disagree that you don't need more than 3. As the article states, if you leverage array-of-structs rather than struct-of-arrays you can use this to "deconstruct" objects without paying the memory usage penalty of struct padding. The 15% wasted RAM in this example is relatively small compared to some real use scenarios; something as common as a 3D vector will often have a whopping 25% space waste.
Other languages allow this as well (and often using such iterations are much faster than zip()ing lists together) but the lack of guarantees and repetitive syntax becomes a pain.
Which means you can implement yours to fit your needs.
> I don't know any language other than Python that shares Python's iterator zip() implementation.
https://docs.rs/itertools/latest/itertools/macro.izip.html
> In implementation
Which is hardly relevant. Python's entire implementation has aims, means, and purpose with no relation to Zig's.
> I also disagree that you don't need more than 3.
Which is not what I wrote.
> As the article states, if you leverage array-of-structs rather than struct-of-arrays you can use this to "deconstruct" objects without paying the memory usage penalty of struct padding.
Sure? And the article uses an example with 3 values.
> The 15% wasted RAM in this example is relatively small compared to some real use scenarios; something as common as a 3D vector will often have a whopping 25% space waste.
It also could hardly be less relevant: it's an issue in an AoS structure because all your objects have that overhead, therefore that's your total overhead.
Here it's 15 or 25% padding in a single value within a stackframe. You're probably wasting more stackframe space due to the compiler not bothering reusing temporally dead locations.
And that's if the compiler reifies the tuple instead of eliding the entire thing.
> Other languages allow this as well
OK?
> (and often using such iterations are much faster than zip()ing lists together)
Until they are not.
Which this doesn't, as zip is an expression and multi-sequence loops aren't.
> https://docs.rs/itertools/latest/itertools/macro.izip.html
External libraries aren't part of a language.
> Which is not what I wrote.
I admit, I read over the "almost" in "you almost never need more than 3".
> It also could hardly be less relevant: it's an issue in an AoS structure because all your objects have that overhead, therefore that's your total overhead.
> Here it's 15 or 25% padding in a single value within a stackframe. You're probably wasting more stackframe space due to the compiler not bothering reusing temporally dead locations.
That's not true: arrays are byte-addressable so inside an array the alignment can be shorter. An array of 121 33-byte values is 3993 bytes in size, an array of 121 usizes is 968 bytes in size, and assuming enums resolve to 32-bit values an array of 121 enums is also 484 bytes in size. There is no overhead here.
This has advantages and disadvantages. Unaligned access is slower in general but in many cases and unaligned array can be faster because of how many of its entries can be loaded into the CPU cache. There's no definite advantage here in terms of CPU performance, but in terms of RAM usage there is.
> Until they are not.
When does a for loop ever become faster than a generator? The values being mapped over are already evaluated, there is no lazy loading+early stopping to take advantage of the generator.
1. It's simpler syntax than reaching for a zip function. I personally like this design because the conceptual load is pretty low as it feels like a natural extension of simple for loops. Eg you could teach someone simple for loops, then later go "hey, you could do this the whole time!"
2. Zig doesn't have support for custom iterators. Zip is doable using the existing metaprogramming features, but it's not as simple. Support for iterators also likely violates Zig's `No hidden control flow.` maxim. Plus I imagine it's a lot easier for the compiler to perform optimizations this way.
Both points combined are related to Zig's design goal for being good at writing code that can run fast on modern CPU architectures. Being able easily to loop over multiple arrays is a good step for making that practical.
Laremere is correct in that there is no "magic" built-in understanding of iterators in the language, i.e. under the hood calling a `.next()` method, without explicitly having to call it. That _would_ violate the no hidden flow control maxim.
However, as Jayschwa points out, Zig's `while` loop will bind the result of its expression (in its own block scope) if it is non-null and otherwise exit the loop. This gives you essentially the same as a for loop that has some language-level knowledge of the iterator pattern, except there is no hidden flow control (I have to explicitly call `next`).
And indeed the Zig standard library is replete with iterators (and in most of the Zig code I write I will will write iterators for my own collections). For example, `mem.split` returns an iterator:
var it = mem.split(...);
// it.next() returns null after we run out of
// split text and the while loop exits
while (it.next()) |substr| {
// In here we have a non-nil substr
}
That's an interesting point: does Zig miss out on some optimisation possibilities with iterators given they are not a language-level construct? I don't know.
Off-topic, but… I very much like this idea, but I think Zig shares one wart with languages like C++: it's impossible to tell syntactically whether `f()` is a direct or indirect call. An indirect branch is a conditional branch, where the condition can be arbitrarily far away in space and time, and it's invisible. Dynamic control flow can be tricky to reason about even when you know it's there.
That is, in a good language you'd replace an explicit but verbose, error-prone and boilerplate for loop with a declaration of the output of said loop. Either through applying a map function or a python-style list comprehension.
(I know this doesn’t matter but I figured the author would appreciate the heads up!)
It's not, because most languages don't have an `else` clause in their for loop (and in my experience with Python that clause is quite confusing so its use is not common).
And a for loop can be executed 0 times, so without a mechanism for a fallback it might not have a value to yield.
I would think that and the similar case where no iteration hits break are solvable by having a for loop return an optional type.
Say like the for loops in racket, or my own loops for guile scheme: https://git.sr.ht/%7Ebjoli/goof-loop
Not sure how I fell about the UB - is it really necessary to optimise away a single length check per loop (not iteration)?
Safety checks can also be enabled or disabled on a scope-by-scope basis if desired.
That said, this likely will be an esoteric case on most modern machines (like x86_64 has 16 regs that can be used for indexes) and I doubt people want to use this for like avr.
Loop macro for parallel/nested iteration featuring a (user-extensible!) vocabulary of clauses:
$ cppawk '
#include <cons.h>
#include <iter.h>
BEGIN {
loop (list(iter0, item, list("alpha", "charlie", "bravo")),
list(iter1, ltr, list("a", "b", "c")),
range(i, 1, 3))
{
print item, ltr, i
}
}'
alpha a 1
charlie b 2
bravo c 3
Preprocessed programs cam be captured, to run on systems that don't have the cppawk script or a preprocessor, and with less startup overhead.
In Haskell, this is called a parallel list comprehension:
[x+y | x <- xs | y <- ys]
> For example, the following zips together two lists:
[ (x, y) | x <- xs | y <- ys ]
For clarity, here's your normal list comprehension (with one pipe) that produces all the combinations instead:
[ (x, y) | x <- xs, y <- ys ]
{-# LANGUAGE ParallelListComp #-}
import Control.Monad (mapM)
elems = [ "water", "earth", "fire", "air" ]
nats = [ "tribes", "kingdom", "nation", "nomads" ]
main = mapM putStrLn
[ show idx ++ " - " ++ e ++ " " ++ n
| e <- elems
| n <- nats
| idx <- [0..]
]
main = forM (zip3 elems nats [0..]) $ \(e, n, idx) ->
putStrLn (show idx ++ " - " ++ e ++ " " ++ n)
main = mapM putStrLn $ do
(e, n, idx) <- zip3 elems nats [0..]
[ show idx ++ " - " ++ e ++ " " ++ n ] for (elems) |x| {
std.debug.print("{} ", .{x});
}
[1] https://github.com/ziglang/zig/blob/f6c934677315665c140151b8...
`.{a, b, c}` is the syntax for an anonymous struct/array/tuple, and a single element still needs to be wrapped in it.
So, if you just type X, you're getting an error about it not being a struct. That's unless X is a struct with one field, where it'll just print that field. I find Zig meta-programming to actually be fairly readable, here's the function that does the formatting: https://github.com/ziglang/zig/blob/master/lib/std/fmt.zig
I think I would go for something expressing ‘default’, but would first look at existing code to see how common this is, and if I decided I wanted this feature, look hard for alternative syntax.
const match: ?usize = for (text, 0..) |x, idx| {
if (x == needle) break idx;
}
I guess they picked this because Python has it, too. https://docs.python.org/3/tutorial/controlflow.html#break-an...:
“Loop statements may have an else clause; it is executed when the loop terminates through exhaustion of the iterable (with for) or when the condition becomes false (with while), but not when the loop is terminated by a break statement.”
import std;
void main()
{
int[] a = [1, 2, 3];
string[] b = ["a", "b", "c"];
foreach (e1, e2; zip(a, b))
{
writeln(e1, ":", e2);
}
}
1:a
2:b
3:cAnyone knows what language Zig was inspired by(if any)? This doesn't look intuitive nor does it resemble any common syntax.
Special ops are formed by adding another symbol onto an operator: wrapping ops are +% -% etc, saturating ops are +| -| etc. This pattern was perhaps inspired by OCaml, which uses +. -. etc for floating point ops. The | for saturating is because "| looks like a wall".
The <op>= syntax is of course from C.
p.* is pointer deference. It was probably chosen for parsing reasons. The * is C for dereference, the . prevents ambiguity with multiplication. It may have been inspired by the postfix p^ from Pascal.
The .fire is just MyEnumType.fire with MyEnumType omitted; the type will be inferred.
for i := 0; i < n; i++ {
fn(foo[i], bar[i])
}
for i := range foo {
fn(foo[i], bar[i])
}
for i := range bar {
fn(foo[i], bar[i])
}
for i, x := range foo {
fn(x, bar[i])
}
for i, y := range bar {
fn(foo[i], y)
}
for _, (x, y) := range (foo, bar) {
fn(x, y)
} func multiLoop[X, Y any](x []X, y []Y, cb func(i int, x X, y Y)) {
if len(x) != len(y) {
panic("invalid slice lengths")
}
for i := 0; i < len(x); i++ {
cb(i, x[i], y[i])
}
}
func foo() {
multiLoop([]int{1, 2, 3}, []string{"a", "b", "c"}, func(i int, x int, y string) {
fmt.Println(i, x, y)
})
}It translates to "increment from one, stopping before you get to five". Ridiculous.
1 increasing UNTIL 5.
It is genuinely recommended (Djikstra onwards) to specify ranges as Start Value (inclusive) Until Final Value (exclusive).
So, I'm assuming zig generally cannot be used with multi-threaded code? Can the underlying arrays not be modified during the whole loop execution?
For slices, length is only known at runtime, but it's immutable once the slice is created, so there's no issue there, either.
No doubt Zig has changed alot and is better than it was only a year or two ago.
Is anyone here willing to say if they have experienced success and satisfaction using Zig? I'm wanting to do some C library interfacing.
Zig is exactly what I want out of a language though, so take my opinion with a grain of salt :)
[0]: https://github.com/natecraddock/zf
What is exactly what you want from a language?
Zig is a very consistent language in syntax and semantics, so there are a small number of features I need to be concerned with. My understanding is that once Zig reaches a stable 1.0 the language will not change. Although there is a lot of churn right now, I appreciate the idea of a language that is simple, and stays simple.
The code is also very readable. I haven't found another language (yet) that I can just open up the standard library source code and understand just about everything. With no hidden control flow I can easily read a function and not have to question what a line of code does. Everything it does is right in front of me.
I also love that Zig is trying to fix many of C's problems. Rather than a global libc allocator, each function that can allocate (by convention) accepts an allocator as an argument. In my projects this has been really great for understanding what code paths allocate, which has made it easy to package my fuzzy finder as an allocation-free Zig module and C library.
Now, if I were working on a project with more critical safety requirements, I might consider a different language. But for most of my personal projects Zig is exactly what I need.
I think that a separate foreach or for-each loop, made for built-in or extended containers could be a nice addition.
Not seeing much value there.
Andrew has a full talk about how the Zig compiler benefits tremendously from DOD:
https://vimeo.com/649009599?embedded=true&source=video_title...
std.debug.print("{} ", .{n});
Thanks!
var i: usize = 0;
while (i < sz) : (i += 1) {
...
}
It did have a foreach-style for loop, as seen in the article though.
EDIT: Nope. I was wrong. This is not a list comprehension.
zip as bs = [(a,b) | a <- as | b <- bs]
When reading zig code you have to stop and think, "wait does this syntax mean zip or direct product?" But when expressed as a function called zip, the meaning is clear.
(Obligatory reminder that zig devs think that sometimes running code inside `if (false)` is a minor bug of no consequence, and after all what are the real motives of anyone pointing it out, eh?)