Is this different from good old `zip`?
1. It's simpler syntax than reaching for a zip function. I personally like this design because the conceptual load is pretty low as it feels like a natural extension of simple for loops. Eg you could teach someone simple for loops, then later go "hey, you could do this the whole time!"
2. Zig doesn't have support for custom iterators. Zip is doable using the existing metaprogramming features, but it's not as simple. Support for iterators also likely violates Zig's `No hidden control flow.` maxim. Plus I imagine it's a lot easier for the compiler to perform optimizations this way.
Both points combined are related to Zig's design goal for being good at writing code that can run fast on modern CPU architectures. Being able easily to loop over multiple arrays is a good step for making that practical.
Laremere is correct in that there is no "magic" built-in understanding of iterators in the language, i.e. under the hood calling a `.next()` method, without explicitly having to call it. That _would_ violate the no hidden flow control maxim.
However, as Jayschwa points out, Zig's `while` loop will bind the result of its expression (in its own block scope) if it is non-null and otherwise exit the loop. This gives you essentially the same as a for loop that has some language-level knowledge of the iterator pattern, except there is no hidden flow control (I have to explicitly call `next`).
And indeed the Zig standard library is replete with iterators (and in most of the Zig code I write I will will write iterators for my own collections). For example, `mem.split` returns an iterator:
var it = mem.split(...);
// it.next() returns null after we run out of
// split text and the while loop exits
while (it.next()) |substr| {
// In here we have a non-nil substr
}
That's an interesting point: does Zig miss out on some optimisation possibilities with iterators given they are not a language-level construct? I don't know.
Off-topic, but… I very much like this idea, but I think Zig shares one wart with languages like C++: it's impossible to tell syntactically whether `f()` is a direct or indirect call. An indirect branch is a conditional branch, where the condition can be arbitrarily far away in space and time, and it's invisible. Dynamic control flow can be tricky to reason about even when you know it's there.
That is, in a good language you'd replace an explicit but verbose, error-prone and boilerplate for loop with a declaration of the output of said loop. Either through applying a map function or a python-style list comprehension.
It's a distributed database for tracking accounts and transfers of amounts of "thing"s between accounts (currency is one example of a "thing"). You might also be interested in our FAQ on why someone would want this [1].
[0] https://github.com/tigerbeetledb/tigerbeetle#quickstart
[1] https://docs.tigerbeetle.com/FAQ/#why-would-i-want-a-dedicat...
And you can model external accounts that have their own confirmation process using our two-phase transfer support.
Yes, exactly. You can think of TigerBeetle as your internal ledger database, where perhaps in the past you might have had to DIY your own ledger with 10 KLOC around SQL.
And to add to what Phil said, you can also use TigerBeetle to track transactions with other parties, since we validate all user data in the transaction—there are only a handful of fields when it comes to double-entry and two-phase transfers between entities running different tech stacks.
The TigerBeetle account/transfer format is meant to be simple to parse, and if you can find user data that would break our state machine, then it's a bug.
Happy to answer more questions!
- LMAX inspired
- Static memory allocation
- Zero copy with Direct I/O
- Zero syscalls with io_uring
- Zero deserialization
- Storage fault tolerance
- Viewstamped Replication consensus protocol
- Flexible Quorums
- Deterministic simulation like FoundationDB
First, because I'm a strong believer in defense-in-depth. Secondly because both disk corruption and network packet corruption happen. Alarmingly often, in fact, if you're operating at large scale.
For example, our deterministic simulation testing does storage fault corruption up to the theoretical limit of f according to our consensus protocol.
Details in our other reply to you.
"This means absolute trust in data read from disk or received from other nodes?"
For example, where most databases will have a network fault model, TigerBeetle also has a storage fault model (https://github.com/tigerbeetledb/tigerbeetle/blob/main/docs/...).
This means that we fully expect the disk to be what we call “near-Byzantine”, i.e. to cause bitrot, or to misdirect or silently ignore read/write I/O, or to simply have faulty hardware or firmware.
Where Jepsen will break most databases with network fault injection, we test TigerBeetle with high levels of storage faults on the read/write path, probably beyond what most systems, or write ahead log designs, or even consensus protocols such as RAFT (cf. “Protocol-Aware Recovery for Consensus-Based Storage” and its analysis of LogCabin), can handle.
For example, most implementations of RAFT and Paxos can fail badly if your disk loses a prepare, because then the stable storage guarantees, that the proofs for these protocols assume, is undermined. Instead, TigerBeetle runs Viewstamped Replication, along with UW-Madison's CTRL protocol (Corruption-Tolerant Replication) and we test our consensus protocol's correctness in the face of unreliable stable storage, using deterministic simulation testing (ala FoundationDB).
Finally, in terms of network fault model, we do end-to-end cryptographic checksumming, because we don't trust TCP checksums with their limited guarantees.
So this is all at the physical storage and network layers.
"Zero deserialization? That sounds rather scary."
* assume a non-Byzantine fault model (that consensus nodes are not malicious),
* run with runtime bounds-checking (and checked arithmetic!) enabled as a fail-safe, plus
* protocol-level checks to ignore invalid data, and
* we only work with fixed-size structs.
* have a simple data model (account and transfer structs),
* validate all fields for semantic errors so that we don't process bad data,
* for example, here's how we validate transfers between accounts: https://github.com/tigerbeetledb/tigerbeetle/blob/d2bd4a6fc240aefe046251382102b9b4f5384b05/src/state_machine.zig#L867-L952.
In our experience, zero-deserialization using fixed-size structs the way we do in TigerBeetle, is simpler than variable length formats, which can be more complicated (imagine a JSON codec), if not more scary.
Oh, nice one. Whenever I speak with people who work on "high reliability" code, they seldom even use fuzz-testing or chaos-testing, which is... well, unsatisfying.
Also, what do you mean by "storage fault"? Is this simulating/injecting silent data corruption or simulating/injecting an error code when writing the data to disk?
> validate all fields for semantic errors so that we don't process bad data,
Ahah, so no deserialization doesn't mean no validation. Gotcha!
> In our experience, zero-deserialization using fixed-size structs the way we do in TigerBeetle, is simpler than variable length formats, which can be more complicated (imagine a JSON codec), if not more scary.
That makes sense, thanks. And yeah, JSON has lots of warts.
Not sure what you mean by variable length. Are you speaking of JSON-style "I have no idea how much data I'll need to read before I can start parsing it" or entropy coding-style "look ma, I'm somehow encoding 17 bits on 3.68 bits"?
Exactly! We focus more on bitrot/misdirection in our simulation testing. We use Antithesis' simulation testing for the latter. We've also tried to design I/O syscall errors away where possible. For example, using O_DSYNC instead of fsync(), so that we can tie errors to I/Os.
> Ahah, so no deserialization doesn't mean no validation. Gotcha!
Well said—they're orthogonal.
> Not sure what you mean by variable length. Are you speaking of JSON-style "I have no idea how much data I'll need to read before I can start parsing it"
Yes, and also where this is internal to the data structure being read, e.g. both variable-length message bodies and variable-length fields.
There's also perhaps an interesting example of how variable-length message bodies can go wrong actually, that we give in the design decisions for our wire protocol, and why we have two checksums, one over the header, and another over the body (instead of one checksum over both!): https://github.com/tigerbeetledb/tigerbeetle/blob/main/docs/...
So, how's the experience of implementing this in Zig?
And we're always learning.
But Zig is the charm. TigerBeetle wouldn't be what it is without it. Comptime has been a gamechanger for us, and the shared philosophy around explicitness and memory efficiency has made everything easier. It's like working with the grain—the std lib is pleasant. I've learned so much also from the community.
My own personal experience has been that I think Andrew has made some truly stunning number of successively brilliant design decisions. I can't fault any. It's all the little things together—seeing this level of conceptual integrity in a language is such a joy.
The advantage of Rust would be a nice and readable stack trace to the crashing method, but a core dump would've included even more information for the person debugging the binary, so I think it ends up quite even.
A reminder for all that have forgotten: UB is the one that can email your local council and submit a request to bulldoze the house you’re in. It is not a free core dump.
It is 100% well-defined behavior to dereference these pointers. It always segfaults, which as Jarred mentioned is a lot like a panic.
Rust evangelists need to be careful because in their zeal they have started to cause subtle errors in the general knowledge of how computers work in young people's minds. Ironically it's a form of memory corruption.
On Windows and Linux this is the first 4KiB so range 0x0000 up to 0x1000, unless large pages are on (then it's even more).
On macOS in x64 this is the entire 4GiB memory space, probably a method to help developers port their 32-bit software to x64. I don't know what the zero page size on ARM is.
If your microcontroller doesn't have this guarantee, you can't make use of this feature.
Remember this gem?
https://kristerw.blogspot.com/2017/09/why-undefined-behavior...
Once you trigger UB, all bets are off and your code could do anything. A segfault just means you spun the roulette wheel, bet it all on red, and got lucky your house wasn't bulldozed.
Zig also uses LLVM under the hood, right? So it's subject to these same semantics. An LLVM pointer value cannot legally contain arbitrary non-null non-pointer integers such as 0x2. That's a dead giveaway of UB. And I doubt the emitted Zig code safety-checks every pointer dereference for a value less than 0x1000 before performing the dereference.
Zig UB is not C UB. There is an entire language built on top of it. Just because something behaves a certain way in C, doesn't mean the same thing is true in Zig. Zig is no longer a code generator for C, it has switched to a self hosted compiler a while back. In fact, the language is rapidly progressing to the point where LLVM is a mere optional dependency.
I don't know the semantics around LLVM pointers. I don't see why 0x2 would be invalid, there are plenty of platforms programmed in C(++) that have a flat memory model. It would be quite painful to have a microcontroller where you can't send data to the output pin because LLVM decided that 2 is invalid (but 0 isn't). I've never seen LLVM complain about invalid dereferencing, though, it always ends up doing what the compiler tells it to do as far as I can tell.
Zig pointers will definitely cause UB but most Zig code shouldn't need them. Slices are actually bound checked and should probably be preferred in most cases of pointer arithmetic. Simple pointers can't be increased or decremented so you need to manually go through @intToPtr if you want to do real pointer arithmetic, which is quite unusable.
I haven't used Zig much so I don't know how many Zig semantics are copies of C semantics and how many are translated by the Zig frontend. However, "this is a bad/undefined thing in C so it must be a bad/undefined thing in Zig" is simply not true.
LLVM has rules about what is legal and what is not legal. If you follow the rules, you get well-defined behavior. It's the same thing in C. You could compile a safe language to C, and as long as you follow the rules of avoiding UB in C, everything is groovy.
Likewise, this is how Zig and other languages such as Rust use LLVM. They play by the rules, and get rewarded by well-defined behavior.
I will return the courtesy, with regards to my interpretation:
> An integer constant other than zero or a pointer value returned from a function not defined within LLVM may be associated with address ranges allocated through mechanisms other than those provided by LLVM. Such ranges shall not overlap with any ranges of addresses allocated by mechanisms provided by LLVM. [2]
[1]: https://llvm.org/docs/LangRef.html
[2]: https://llvm.org/docs/LangRef.html#pointer-aliasing-rules
- Any memory access must be done through a pointer value associated with an address range of the memory access, otherwise the behavior is undefined.
- A null pointer in the default address-space is associated with no address.
A null pointer (0x0) is associated with no address, therefore it has no address range. So if you do attempt a memory access (dereference), the behavior is undefined. QED. A naive translation to assembly would indeed segfault on a modern OS, but LLVM's optimizations are free to assume that code path is unreachable and do anything else.
Once the program is in this state, a bug of some kind is unavoidable. I don't take issue with that - what I take issue with is your claim that this behavior is well-defined, because it definitely is not. It would be equally valid for a null dereference to corrupt your program state or wipe your hard disk.
In Rust for example, derefencing a raw pointer is unsafe - because that pointer could have a value of 0x2 - which would result in undefined behavior according to LLVM.
tbh I'm surprised any of this is even up for debate. If you google "is segfault undefined behavior" you'll get 100 results telling you yes, yes it is.
I'm not sure how I can connect the dots any more clearly. Like gggggp said, it's baffling to see the creator of a popular language sweep the nasal demons under the rug and pretend that certain undefined behavior is guaranteed.
Calling such segfaults "safe" or "well-defined" is setting your users up for disappointment and CVEs, because a "well-defined" result is axiomatically impossible in the presence of undefined behavior. It's subtle, and if we were talking about a Java competitor maybe I could forgive the mistake. But if you're writing a low-level language it's important to understand how this stuff works. Ironically, he spread misinformation in the very post where he accused Rust evangelists of the same.
This thread is long dead and continuing the discussion seems futile, so I'll just leave it at that.
*excluding something silly like `raise(SIGSEGV)`
Do the docs actually define exactly which mechanisms external to LLVM count as allocating address ranges and which do not? It's possible that calling mmap and passing PROT_NONE does not count, for example.
I don't believe PROT_NONE suffices. The address needs to be accessible, not merely mapped. If reading through a pointer, the address must be readable. If writing through a pointer, the address must be writeable. This is why writing to a string constant is undefined behavior, even though reading would be fine.
Another issue is alignment. If you read from a `*const i32` with unaligned pointer value 0x2, the optimizer is free to assume that code path is unreachable and, you guessed it, bulldoze your house. If you get a segfault from reading an `i32` from address 0x2, you've already hit UB and spun the roulette wheel.
In theory the emitted code could check pointers for alignment and validity (in whatever platform-specific way) before accessing them, and simulate a segfault if not. Such checks would serve as optimization barriers in LLVM, and prevent these instances of UB. Of course Zig's current ReleaseSafe doesn't do this, and I think it would be silly if it did. But that's the only way you could accurately call segfaults "well-defined".
0x2 is a perfectly valid pointer value, it just happens to never be a good virtual memory address on modern systems where virtual memory is setup by the usual OSs, hence the fact that you can rely on it segfaulting.
Then I guess it could be a language guarantee if Zig only supports/targets those platforms. However, considering how low-level Zig is, I doubt that that is the case.
It isn't, in the general case. But JavaScript engines do some dark magic with pointer packing / NaN boxing as a performance optimization (most things in the VM are single words, passing around a single word is usually way cheaper than full unboxing), and I suspect bun in occasionally running into issues where it gets returned something from the JS engine which it thinks is a pointer but actually it's a packed, special value. This is a logic error, that turns into a weird memory issue at the abi boundary, not a memory safety issue.
Is that guaranteed by the language semantics, or could it possibly change at some point in the future? If it's the latter, then yes, it is very much Undefined Behavior, and not guaranteed to segfault before opening the door for potential exploits.
Not on every architecture, not in LLVM (even if well-defined on the underlying architecture), and not in C (even if well-defined in the underlying compiler backend).
TL;DR: Zig injects checks and aborts the program at runtime unless you specify that you wish to ignore the problem. This can be done explicitly within the code or by compiling under a build mode that ignores checks (unless specified manually).
Programs compiled as Debug and ReleaseSafe will terminate at runtime if UB is triggered. Compiling for ReleaseSmall and ReleaseFast will cause traditional C-style UB. If you care about your program doing what it's supposed to do, you use ReleaseSafe. Doing Release[Fast|Small] will do something similar to -O3 in other languages, which will often change behaviour.
Note, however, that you can compile your code under "just allow UB and see what happens" mode but still benefit from checked UB by setting @setRuntimeSafety(true); this will introduce the assertions despite the unsafe build modes you may specify.
It's like introducing a C++ compiler flag* telling the compiler "ignore exceptions and just continue". You know you're in for a bad time the moment you specify it, but it makes your program blazingly fast because it greatly reduces the amount of code to generate/checks to execute.
The main advantage of checked UB is that well-tested code can make use of the unchecked nature of these features for speed without having length check code blocks that need to be wrapped in debug #ifdefs or similar. Assuming you don't run test builds with checks enabled (and why wouldn't you) you'd catch these problems in your build pipeline.
This is different from the normal way of working with C and friends, where UB remains in debug/-O1 builds but just acts a little differently. Some compilers will insert breakpoints, others will ignore the problem like in release mode, nobody knows what will happen and your compiler can't detect this problem for you.
* note that -fno-exceptions exists, but that aborts the program rather than let it continue.
I can enable lightweight assertions in libstdc++ and libc++ and it makes C++ safer, but not in any way "safe". There are some flags that can be enabled to trap on some language UB too, without bringing in the heavy weight sanitizers.
So, if I read this correctly, barring the simple cases that Zig can detect at compile-time, this means that whether it's a UB (in the C++ definition of the term) depends on the flags specified by the author of the library and the person who compiles the final binary.
That's definitely much better than C++ UB. Still a bit scary, though.
I thought it was defined as an OOB access in non-safe modes, or a panic in safe-modes.
I do wonder about performance, though, as multiple array derefences may not be captured well by the L1 cache like a well-rounded struct might.
An L1 cache line is often 64 bytes long, enough to fit one of the "monster" example structs but never two. Performance in real life scenarios may actually increase if these structs are padded with an additional 16 bytes so none of the structs are on a cache line boundary.
So… zip?
Python’s does that, and for most other langages you could use overloading, basic macros, or traits trickery to get there if you really wanted to support unreasonable widths (IME you almost never need more than 3, and combining two zip/2 works fine then).
In implementation, Python's zip will return a generator that is iterated over using the iterator functionality, while Zig's .zip is compiled as a loop. Python's iteration may be turned into a loop, it may be interpreted, or it may be turned into some other kind of bytecode, who knows. The standard cpython implementation is much more complex, though: https://github.com/python/cpython/blob/main/Python/bltinmodu...
Concatenating zip()s is an unnecessarily complex solution, both in terms of syntax and in code generated. In Python this may not matter because it's a relatively slow programming language in general (the language often being "glue between fast C methods"), but in Zig this can easily become untennable.
I also disagree that you don't need more than 3. As the article states, if you leverage array-of-structs rather than struct-of-arrays you can use this to "deconstruct" objects without paying the memory usage penalty of struct padding. The 15% wasted RAM in this example is relatively small compared to some real use scenarios; something as common as a 3D vector will often have a whopping 25% space waste.
Other languages allow this as well (and often using such iterations are much faster than zip()ing lists together) but the lack of guarantees and repetitive syntax becomes a pain.
Which means you can implement yours to fit your needs.
> I don't know any language other than Python that shares Python's iterator zip() implementation.
https://docs.rs/itertools/latest/itertools/macro.izip.html
> In implementation
Which is hardly relevant. Python's entire implementation has aims, means, and purpose with no relation to Zig's.
> I also disagree that you don't need more than 3.
Which is not what I wrote.
> As the article states, if you leverage array-of-structs rather than struct-of-arrays you can use this to "deconstruct" objects without paying the memory usage penalty of struct padding.
Sure? And the article uses an example with 3 values.
> The 15% wasted RAM in this example is relatively small compared to some real use scenarios; something as common as a 3D vector will often have a whopping 25% space waste.
It also could hardly be less relevant: it's an issue in an AoS structure because all your objects have that overhead, therefore that's your total overhead.
Here it's 15 or 25% padding in a single value within a stackframe. You're probably wasting more stackframe space due to the compiler not bothering reusing temporally dead locations.
And that's if the compiler reifies the tuple instead of eliding the entire thing.
> Other languages allow this as well
OK?
> (and often using such iterations are much faster than zip()ing lists together)
Until they are not.
Which this doesn't, as zip is an expression and multi-sequence loops aren't.
> https://docs.rs/itertools/latest/itertools/macro.izip.html
External libraries aren't part of a language.
> Which is not what I wrote.
I admit, I read over the "almost" in "you almost never need more than 3".
> It also could hardly be less relevant: it's an issue in an AoS structure because all your objects have that overhead, therefore that's your total overhead.
> Here it's 15 or 25% padding in a single value within a stackframe. You're probably wasting more stackframe space due to the compiler not bothering reusing temporally dead locations.
That's not true: arrays are byte-addressable so inside an array the alignment can be shorter. An array of 121 33-byte values is 3993 bytes in size, an array of 121 usizes is 968 bytes in size, and assuming enums resolve to 32-bit values an array of 121 enums is also 484 bytes in size. There is no overhead here.
This has advantages and disadvantages. Unaligned access is slower in general but in many cases and unaligned array can be faster because of how many of its entries can be loaded into the CPU cache. There's no definite advantage here in terms of CPU performance, but in terms of RAM usage there is.
> Until they are not.
When does a for loop ever become faster than a generator? The values being mapped over are already evaluated, there is no lazy loading+early stopping to take advantage of the generator.
Unlike Python, you can't pass a zip generator around. It's just a for-loop. While zig loops are expressions, they only return a single value.
Does it, actually? Does Zig's built-in pseudo-zip outperform Rust's? Or C++23's?
> It's just a for-loop.
Except it's not "just" a for loop, it's a weird special case for a for loop. And one which is actively dangerous too.
You could also argue that a for loop which can only iterate over a _single_ sequence is a special case of a multi-sequence for loop.
> And one which is actively dangerous too.
In safe builds (ReleaseSafe, Debug) it will cause a controlled panic if the sequences are not of the same size. Most likely it's a logical bug if you iterate over two sequences of different sizes. In ReleaseFast the compiler will make assumptions to improve performance. If it's very important for your code you can force a certain code block to always have runtime safety. Yes, there are trade-offs, but I don't feel it's _unreasonable_.
When a "for loop" in virtually every language is not multi-sequence, not really. People expect a "for loop" to be a certain kind of thing.