Can you expand on this ? I'm asking out of curiosity because I want to learn a "system" programming language (for whatever definition there is to this term). So far I briefly tried Rust and Nim and found the former more difficult to read. I know nothing about Zig, how would you place it between these two ?
* zig has native support for arbitrary sized integers. In Rust I used to do bitshifts, Now I just have a packed struct of u3/u5/u7 whatever (see `src/pci/pci.zig`). Of course Rust has a bitflags crate but I didn't find it handy, this is a case where native support vs library support means a world of difference.
* zig has native support for freestanding target. I used to have to build with Xargo for cross compiling to a custom target, I also was forced to `[nostd]`, and some features I was using forced me to use nightly Rust. In zig I have a simple `build.zig` and a simple `linker.ld` script, just works.
* zig has nicer pointer handling. A lot of kernel programming is stuff that Rust considers unsafe anyway. It's not uncommon to have to write lines like `unsafe { (ptr as const u8) }` to deref a pointer in Rust, which is a pain because this kind of thing happens all of the time. Also you have to play around with mut a lot like this: `unsafe { &mut (address as mut _)`. It just felt wrong a lot of the time, where in zig you have either `const` or `var` and that's the end of it.
* zig is really fun to write! this is something that comes up often in the community, after years of C it's just very refreshing.
Some things zig is missing:
* Package manager, this is coming soon. [1]
* Missing documentation for inline assembly (I think this part is going to get overhauled, as Andrew Kelley is writing an assembler in zig atm [2]).
I don't know Nim, but I believe it has a garbage collector so it could be tricky to use for kernel programming.
[0] https://github.com/jzck/kernel-rs
Arbitrary sized integers do sound very convenient, although I guess the counterargument would be that they simply can't map to machine types so you'll have to have magic behind the scenes which may not make complete sense in low level languages. Still, C has bitfields in structs which are sort of the same thing so why not.
Pointer handling however I'm not sold on. I'd argue that the very cumbersome pointer handling in Rust is a feature, not a bug. It's explicitly discouraged, and for good reasons. Map your pointer into a clean and safe reference, slice or wrapper object ASAP and leave raw pointers to the messy details of C FFIs and ultra low-level code. If you end up having to mess with raw pointers everywhere in your codebase you're doing it wrong, as far as Rust is concerned.
I think this is reasonable even for kernel code. After all typically you wouldn't dereference register pointers directly in C code because of volatility issues and making it obvious that you're accessing hardware and not RAM (readl/writel or similar), so having a safe wrapper adds no overhead in my experience.
* I don't think that low level code needs to be "messy", and
* I don't think that adding an abstraction on top solves anything complexity wise.
For memory paging I used to have a whole library in Rust that handled hierarchy of page tables with very abstract pagetable classes and interfaces [0]. Now I just have 80 lines of zig that handles everything [1]. I much prefer the latter.
> I think this is reasonable even for kernel code.
It's not, for some of the reasons that Linus doesn't want C++ code in the kernel [2]
[0] https://github.com/gz/rust-x86/blob/28d5839933973e0b639ef354...
[1] https://git.sr.ht/~jzck/kernel/tree/master/src/arch/x86/pagi...
I don't think what Linus wrote in 2004 on C++ has much relevance for 2020 Rust, especially for a new kernel that doesn't have a ton of contributors. Most of his complaints seem to be on how C++ abstractions can make it hard to review unknown code and how 2004 C++ code bases often had extremely messy code. Rust IMO doesn't suffer from those problems near as much as 2004 C++ did.
Also, Google is using Rust for part of their new Fuchsia kernel so at least they think Rust has something good to offer kernel dev.
No, it's written in C++[1]. They're very different languages.
That said, Google seems to think C++ has something to offer to all kinds of development. It seems to work for them, but they are a heavily C++ shop. I wouldn't read too much out of their use of C++ anywhere; it'd be more surprising and interesting to see them use anything else.
[1]: https://fuchsia.googlesource.com/fuchsia/+/master/zircon/ker...
[0]: https://fuchsia.dev/fuchsia-src/concepts/kernel/zx_and_lk
Output:
Test [1/1] test "bit field access"...
*align(:3:1) u3
All 1 tests passed.
- fn deref(ptr: var) u3 {
+ fn deref(ptr: *u3) u3 {
test.zig:13:23: error: expected type '*u3', found '*align(:3:1) u3'
assert(deref(&data.b) == 2);
^This is the main risk with bitfields in C, it makes reads and writes that appear to be independent actually access the same memory cell behind the scenes. I'm not convinced by the idea of adding even more magic on top of it to be honest, it is true that packing and unpacking bitfields is annoying and requires some boilerplate but in the end that is how it the hardware does it anyway.
Have a look at Project Oberon, Singularity, Interlisp-D, Smalltalk, Modula-3 Topaz, Blue Bottle (AOS), CosmosOS, lilith (Crystal variant) given that their source is available.
> I don't know Nim, but I believe it has a garbage collector so it could be tricky to use for kernel programming.
You're right. Still good for libraries though (or apps, but that may be outside of "system").
This is what I use for my own multithreading runtime in Nim and the memory subsystem makes it faster and more robust than any runtime (including OpenMP and Intel TBB) that I've been benchmarking against, see memory subsystem details here: https://github.com/mratsim/weave/tree/master/weave/memory
Example of atomic refcounting in this PR here: https://github.com/mratsim/weave/blob/025387510/weave/dataty...
Also one important thing, Nim current GC is based on TLSF (http://www.gii.upv.es/tlsf/) which is a memory reclamation scheme for real-time system, it provides provably bounded O(1) allocations. You can tune Nim GC with max-pauses for latency critical applications.
Also, the new ARC memory manager replaces GC and can run in a kernel.
As soon as you use sequences or strings or async you depend on the GC.
You can however compile with gc:destructors or gc:arc so that those are managed by RAII.
I meant that new ARC GC, that will replace the current one, can be used for the kernel.
It's still a GC, technically, but, quoting Araq on ARC:
Nim is getting the "one GC to rule them all". However calling it a GC doesn't do it justice, it's plain old reference counting with optimizations thanks to move semantics.
otherwise you should be able to use something like destructors eventually
Then I guess you would use new/dealloc
https://github.com/vitreo12/omni
I don't know enough to comment but it may be useful to look at things in the wild. This project also heavily relies on calling in C to interface with environments and the SuperCollider scsynth.
But you can also think of Nim as macro-able higher level C and write it like that (but indeed you probably still need this minimal allocation support)
That's really cool.
I apologize if this is a naive question. Might you or someone else elaborate on where arbitrary sized integers are used or necessary in kernel programming? Don't they all ultimately need to get padded out for the CPU registers to work with them anyway?
Proof (and they have a policy to only use unsafe when not otherwise possible):
https://github.com/redox-os/kernel/search?q=unsafe&unscoped_...
saying "if you write your kernel in rust, it will be memory safe" is a little goofy, because writing kernels in rust requires you to drop into unsafe blocks way too often.
Unsafe is definitely used, but it does not mean what I think is being implied, which is that it makes the software unsafe.
Source code is available.
I personally think the Rust community has missed the biggest advantage of unsafe, and that's tooling. Imagine an IDE that could highlight and warn when touching state that is accessed in unsafe blocks, or when state being accessed in unsafe blocks isn't being properly tested.
Or tools that kick off notification for a required code review because code that touches state used by an unsafe block got changed.
To me, these sorts of things will have a far greater effect on security and stability than just the existence of the unsafe block in code.
The rest are just plugins and tools that look for said blocks and “do stuff”. I don’t see why these tools should be part of the language.
Another poster responded to me stating they are starting to work on those tools, so it looks like the rust community is starting to come around.
It's just that I've seen a lot of people tout the safety of rust as if the unsafe keyword was a panacea that automatically prevented problems. That attitude is really what I was responding to.
The Rust community can be a bit... fanatical, lets say. I've seen so many people argue that the unsafe keyword by itself makes rust so much safer than alternatives, when really safe code can cause unsafe code to explode. So you end up writing modules to protect the unsafe code, which is what you do in any other language as well.
Which means the unsafe keyword is valuable, just not nearly as valuable as I've seen a lot of people claim.
Now the unsafe keyword combined with the sort of tooling I've mentioned? That to me is a killer combination. Unsafe isn't a silver bullet, it still requires work, but it enables tools to make that work immensely easier to deal with.
I've talked about this at length before, but just because you have 'less' `unsafe` code does not mean you code is any better off. Very few operations are actually `unsafe`, and you're still not allowed to break any of the (poorly or not documented) constraints that Rust requires even in `unsafe` code.
Point being, unless you can make "safe" abstractions around your `unsafe` code that cannot be broken (Which is largely what you do in any language anyway, when you can...), the distinction between "safe" and "unsafe" is thin, because the parts of the code you declare `unsafe` are not the parts that are actually going to have the bugs. And because of the fact that the constraints you are required to fulfill in `unsafe` code is unclear, there's little way to guarantee your code is broken in some subtle way the optimizer might jump on (either now, or in a later release).
And to be clear, I like Rust, but `unsafe` is poorly thought-out IMO.
Sure, but now we're back to the issue that it's unclear what constraints `unsafe` code actually has to hold, meaning ensuring your abstraction is safe is just about impossible to do with certainty (And OS kernel code is definitely going to hit on a lot of those corner cases). You may think you have a safe interface, only to find out later you're relying on internal details that aren't guaranteed to stay the same between compiler versions or during optimizations.
With that said, while I agree with what you're proposing about marking surrounding code "unsafe", it leads to lots of strange cases and most people get it wrong or will even disagree with this proposal completely. For example, it can lead to cases where you mark a function `unsafe` even though it contains nothing but "safe" code. And at that point, it's up to you to determine if something is actually "safe" or "unsafe", meaning the markings of "safe" and "unsafe" just become arbitrary choices based on what you think "unsafe" means, rather than marking something that actually does one of the "unsafe" operations.
One of the best examples is pointer arithmetic. It's explicitly a safe operation, but it's also the first things most people would identify as being "unsafe", even though it is dereferencing that is the "unsafe" operation. Ex. You could easily write slice::from_raw_parts without using any `unsafe` code at all, it just puts a pointer and length together into a structure (it doesn't even need to do arithmetic!). It's only marked `unsafe` because it can break other pieces of "safe" code that it doesn't use, but it itself is 100% safe. You could just as easily argue the "other code" should be the `unsafe` code, since it's what will actually break if you use it incorrectly.
Perhaps the biggest annoyance I have is that the official Rust documentation pretty much goes against the idea you presented, saying
> People are fallible, and mistakes will happen, but by requiring these four unsafe operations to be inside blocks annotated with unsafe you’ll know that any errors related to memory safety must be within an unsafe block. Keep unsafe blocks small; you’ll be thankful later when you investigate memory bugs.
Which is just incorrect - memory safety issues are likely to be due to the "safe" code surrounding your unsafe code - when you're writing C code, the bug isn't that you dereferenced NULL or an OOB pointer, the bug is the code that gave you that pointer in the first place, and in Rust that code is likely to all be "safe". Point being, most of the advice on keeping your unsafe blocks small just leads to people making silly APIs that can be broken via the safe API they wrap (Even in weird ways, like calling a method containing only safe code), and unfortunately there are lots of subtle ways you can unintentionally break you Rust code that most people aren't going to have any idea about.
The Rust Nomicon actually documents these constraints for each 'unsafe' operation. Code that can ensure that these constraints hold can be regarded as 'safe' and rely on an unsafe{ } block. Code that can't, should be marked unsafe and document its own constraints in turn.
> You may think you have a safe interface, only to find out later you're relying on internal details
If you're relying on internal details, you're most likely doing it wrong, even by the standards of 'unsafe'. There are ways to nail down these details where required, at which point they're not even "internal" details anymore, but this has to be opted-into explicitly, for sensible reasons.
> It's only marked `unsafe` because it can break other pieces of "safe" code that it doesn't use, but it itself is 100% safe. You could just as easily argue the "other code" should be the `unsafe` code, since it's what will actually break if you use it incorrectly.
No, "other code" should not be marked unsafe because this would mean that relying on slices is inherently unsafe. Which is silly; the whole point of a "slice" type, contrasted with its "raw parts", is in the guarantees it provides. This is why slice::from_raw_parts is unsafe: not because if what it does, but what it states about the result.
> Which is just incorrect - memory safety issues are likely to be due to the "safe" code surrounding your unsafe code
This is a matter of how you assign "blame" for a memory safety error. It's definitely true however, that the memory unsafe operations play a key role, and I assume that's the point that these docs are making. I agree that people shouldn't be creating faulty abstractions around "unsafe" blocks, but the reason we can even identify this as an issue is because we have "unsafe" blocks in the first place!
It's not that simple, what I'm getting at is that 'unsafe' code is still required to meet all the constraints that 'safe' code has to meet, even though those constraints are not fully documented (Because in general, in 'safe' code there are lots of things that are impossible to do but not explicitly UB), and they have changed them in significant ways in the past. You can see an incomplete list of them here[0]. There has been work to nail these details down for years, with the most recent being this one[1], but in general I would argue there hasn't been much progress since I first looked into it years ago now. Fun fact, the first Rust issue I remember reading that touched on this (and evolved into basically everything going on today) is this[4] one, which was made exactly 4 years ago today (And I mean exact!).
The 'Safety' sections for 'unsafe' operations are actually just extra requirements for those particular API on top that you also have to keep in addition to whatever safe Rust constraints apply. And I would argue in general they're not actually exhaustive, and they're mostly just a "best guess" - it's not a breaking change to introduce or realize there are more constraints. Ex. A little more then a year ago, they added the info that a slice can't be larger than `isize::MAX`[2], and a few months ago was this[3] one, where they added that the pointer length must be from a single allocation.
My point is that when you're doing something like writing an OS kernel, you end up running into these types of issues and corner-cases because Ex. They determine what your allocator is allowed to produce, or if some particular pointer can be validly put into a slice.
> No, "other code" should not be marked unsafe because this would mean that relying on slices is inherently unsafe. Which is silly; the whole point of a "slice" type, contrasted with its "raw parts", is in the guarantees it provides. This is why slice::from_raw_parts is unsafe: not because if what it does, but what it states about the result.
Sure, my point is that this is at best unclear, and I would argue wildly misunderstood. We've basically made two definitions of 'unsafe' - one for code that has to perform one of the five 'unsafe' operations (Which is a clearly defined 'yes' or 'no', and also the one described by the Rust docs), and one for code that can potentially break some type of invariant the will potentially cause UB somewhere else in the program (even if the code in question is completely safe and can't on its own cause any problems).
The problem is that Rust only guarantees the type of safety that it does if you do the second, but yet lots of people (I would almost argue most...) assume the first is sufficient and 'unsafe' by itself ensures all your memory bugs are in `unsafe` code. The Rust docs even describes safe vs. unsafe as two completely different languages (which they are) which brings into question why you would write `unsafe` code when you don't even need any of the `unsafe` operations. You arguably want a separate keyword for this - one that marks a function `unsafe`, but doesn't actually allow you to perform any of the `unsafe` operations in it.
[0] https://doc.rust-lang.org/nomicon/what-unsafe-does.html [1] https://github.com/rust-lang/unsafe-code-guidelines [2] https://github.com/rust-lang/rust/commit/1975b8d21b21d5ede54... [3] https://github.com/rust-lang/rust/commit/1a254e4f434e0cbb9ff... [4] https://github.com/rust-lang/rfcs/issues/1447
These breaking changes are allowed precisely because they're meant to address soundness concerns. Even if at any given time they're just a "best guess" of what's actually needed, that's still wildly better than what e.g. C/C++ do, which is just to not guess at all. Even wrt. formal memory models, perhaps the most complex issue among the ones you mention here, C/C++ only got an attempt at a workable memory model with the 201x standards (C11, etc.). It's not unreasonable to expect that Rust may ultimately do better than that.
> It's not unreasonable to expect that Rust may ultimately do better than that.
But, when? Rust is almost 10 years old, and questions about this stuff has been posed for years now - I was having these same conversations over two years ago. Last year's roadmap included working on the 'unsafe guidelines' I posted, and as an outsider looking in it's unclear to me how much progress has actually being made. Don't get me wrong - they've made a fair amount of progress, but there's still a lot to be done.
My big concern (which I don't consider to be unfounded) is that because there aren't that many big users of Rust actually developing really low-level stuff, the work on defining it isn't getting done. But to me it feels like a self fulfilling prophecy - a project like the Linux Kernel isn't going to end-up using Rust if there's big issues with connecting what they're doing now to Rust idioms (Ignoring the LLVM thing...), even though when Rust was first maturing it was being billed as the replacement for C (and still is).
When that said, my concerns are more with the state of `unsafe` overall, the memory model is only one part of that (though it somewhat spurred conversions about the other issues).
I bet that the pre-C11 memory semantics on Linux kernel aren't the same as e.g. on HP-UX kernel, across all supported hardware platforms.
It is also ironic that Java and .NET did it before C and C++, with their models being adopted as starting point.
> I bet that the pre-C11 memory semantics on Linux kernel aren't the same as e.g. on HP-UX kernel, across all supported hardware platforms.
Yeah, but if they both work, why does it matter? There exists more than one valid possible memory model. The point I was making is that with C, it has been possible to define valid semantics well before C11 because compilers and the language give enough guarantees already.
To that point, while I focused on atomics (Since atomics and threading are largely what was added in C11), the bulk of the actual memory model that made that possible was defined well before then. Strict-aliasing was a thing in C89 (Though I'm not sure compilers enforced it at that point) and is probably the only real notable "gotcha", and `gcc` and `clang` let you turn it off outright (And if you're not doing weird kernel stuff, it generally is easy to obey).
`gcc` and `clang` also have lots of documentation on the extra guarantees they give, such as very well documented inline assembly, type-punning via `union`, lots of attributes for giving the compiler extra information, etc.
Compared to what Rust offers right now, there is no comparison. With Rust, the aliasing model (Which `unsafe` code is not allowed to break) is still unknown, and a lot of the nitty/gritty details like the stuff I listed for `from_raw_parts` are simply leaky implementation details they're inheriting unintentionally from LLVM (And effectively from C - C is where things like "not allowed to go past one past the end of an allocated block" restriction comes from, along with a host of other things).
That said, for userspace, pthreads already gives enough guarantees on ordering via mutex's that there aren't really any problems unless you're trying to do atomics in userspace, which before C11 would have been hard to do portability. And the rest of the things I was talking about like the aliasing model are defined as part of the C standard (C89 and C99), so it's always the same regardless of the compiler (ignoring possible bugs).
It could be argued that's not a huge value gain since you're so far down the stack, but that's the value of the unsafe keyword.
I see two advantages to explicitly marking code as unsafe:
(1) You are very clearly marking the areas of code that are most suspect
(2) You can still have safe code outside of those “unsafe” blocks
You can also go further like I did here:
https://news.ycombinator.com/item?id=21840431
You're still worrying about a smaller portion of your code.
But I've always had a mental note to go back once it got a bit more stable because I really do like the ideas behind it.
It sounds to me like it's definitely getting there and that's awesome.
If you mean that you don't know any right now, just learn C. This isn't web programming, where everyone is always hopping to the latest fads. C is the lingua franca, the sine qua non of systems programming. It will be a long time before that changes. The C machine model is what everyone is working with anyway. C is small, despite some devious and fun corners of it, and mostly just exposes you to that model and way of interacting with the machine.
You can’t even appreciate the new systems languages unless you have a firm grasp of C and systems programming. They are all a response to C, addendum and proposed evolutions to it.
- no access to carry/overflow flag in registers making it a chore to write bigint libraries
- no way to guarantee emitting add with carry or sub with borrow even though those are simple instructions and some architecture (6502) don't even provide a normal add/sub
- need to drop down to assembly to implement resumable function/fibers to avoid the syscall costs of ucontext/setjmp/longjmp
- no access to hardware counters like RDTSC
- no way to get the CPU frequency in a portable and reliable way
- no portable threadID function and then, those that are available (pthread_self, and Windows') rely on expensive syscalls.
- no way to do CPU feature detection (SSE, AVX, ARM Neon, ...)
- no way to control emission of common intrinsics like popcount, bit-scan-reverse, lowest bit isolation in a portable way.
The portable assembly narrative breaks down rapidly when you actually need specific code to be emitted.
> - no access to carry/overflow flag in registers making it a chore to write bigint libraries
C is portable assembly: so it provides access to a minimum common set of features which allow the code to be portable:
good luck trying to use the carry flag on the RISC-V..
Is there any way to have those concepts be portable across ISAs/architectures. I don’t think those types of ops are exposed in LLVM-IR or GCC’s various intermediate levels (but could be wrong). I’d love to be able to investigate a language that enables a level of hardware access possible in a portable manner.
Which syscalls would be involved in setjmp/longjmp?
And for longjmp, Apple source for example https://opensource.apple.com/source/Libc/Libc-262/i386/sys/s.... It does seem like glibc impl doesn't involve syscalls: https://groups.google.com/forum/#!topic/comp.unix.programmer...
(FreeBSD provides the non-portable _setjmp/_longjmp, which do not preserve signal mask state and avoid the syscall. There are also the POSIX sigsetjmp/siglongjmp, which with savesigs=0 may bypass the syscalls as well. Both FreeBSD and Linux provide the POSIX routines.)
The part where Rust can get really messy is when you involve generic programing and traits, especially when lifetimes are involved, but I couldn't find any obviously generic code in this codebase.
EDIT: after digging a bit deeper into Zig it looks like it uses C++'s style duck typing for metaprograming instead of a trait-based approach like Rust? It definitely removes some overhead to writing generic code but I'm not sure about readability and maintainability... But I'm going to stop here because I'm about to go full fanboy for Rust.
The main difference between Zig and Rust is a huge disparity in language complexity. Zig is a language that can be fully learned in a day or two. Rust has the same philosophy of "zero-cost abstraction" as C++, i.e. spending a lot of complexity budget to make a low-abstraction language appear as if it has high abstraction. Zig, like C, does not try to give the illusion of abstraction.
There is also the difference in their approach to safety, but that's a complicated subject that ultimately boils down to an empirical question -- which approach is safer? -- which we don't have the requisite data to answer.
We do have the requisite data to answer whether preventing use-after-free is better than not preventing it.
You can argue (not successfully, in my opinion) that it's not worth the loss in productivity to prevent UAF and other memory safety issues, but it's impossible to argue that not trying to prevent UAF is somehow safer.
I expect Zig will prevent use-after-free. It will be sound for safe code and unsound for unsafe code (by turning this on only in debug mode for testing).
> but it's impossible to argue that not trying to prevent UAF is somehow safer.
First, see above. Second, it is not only possible but even reasonable to argue that not trying to completely eliminate a certain error is very much safer. The reason is that soundness has a non-trivial cost which can very often be traded for an unsound reduction in a larger class of bugs. As an example, instead of soundly eliminating bugs of kind A, reducing bugs of kinds A, B and C -- for a similar cost -- may well be safer.
There has been little evidence to settle whether sound elimination of bugs results in more correctness than unsound reduction of bugs or vice-versa, and it's a subject of debate in software correctness research.
> As an example, instead of soundly eliminating bugs of kind A, reducing bugs of kinds A, B and C -- for a similar cost -- may well be safer.
Hasn't this essentially been what C++ has been trying for memory safety for decades, without success? The C++ approach has been "smart pointers are good enough, and they prevent several other problems too", and the experience of web browsers (among others) has pretty much definitively shown: no, they really aren't. For memory safety, I would not bet on this approach.
I wouldn't say Zig exists at all right now, but just as it strives to one day be production-ready, it strives to prevent use-after-free. Safety is a stated goal for the language.
> Hasn't this essentially been what C++ has been trying for memory safety for decades, without success?
No. I'm talking about a mechanism that can detect various errors at runtime, and is turned on or off for various pieces of code and/or for all code at various stages of development. Rust, BTW, doesn't entirely guarantee memory-safety, either, when any unproven unsafe code is used, and even when it isn't (e.g., have you proven LLVM's correctness?). We always make some compromises on soundness; the question is where the sweet-spots are.
Software correctness is one area where there are no easy answers and very few obvious ones.
What you are describing exists: ASan. We have a pretty good answer to the question "is ASan sufficient to prevent memory safety problems in practice": "no, not really".
> Rust, BTW, doesn't entirely guarantee memory-safety, either, when any unproven unsafe code is used, and even when it isn't (e.g., have you proven LLVM's correctness?). We always make some compromises on soundness; the question is where the sweet-spots are.
Empirically, Rust's approach has resulted in far fewer memory safety problems than previous approaches like smart pointers and ASan, with only garbage collectors (and restrictive languages with no allocation at all) having similar success in practice. Notice that the working approaches have something important in common: a strong system that, given certain assumptions, guarantees the lack of memory safety problems. Even though those assumptions are never quite satisfied in practice, empirically having those theoretical guarantees seems important. It separates systems that drastically reduce safety problems, such as Rust and GC languages, from those that do so less well, such as ASan and smart pointers. This is why I'm so skeptical of just piling on more mitigations: they're helpful, but we've been piling on mitigations for decades and UAF (for instance) is still a big a problem as ever.
That is not the question we're interested in answering, and elimination of all memory errors is no one's ultimate goal, certainly not at any cost. By definition, unsound techniques will let some errors through. The question is which approach leads to an overall safer program for a given effort, and soundness (of properties of interest) always comes at a cost.
(also, Zig catches various overflow errors better than ASan)
> Empirically, Rust's approach has resulted in far fewer memory safety problems than previous approaches like smart pointers
I don't doubt that, and if minimization of memory errors was programmers' primary concern (even in the scope of program correctness or even just security), there would be little doubt that Rust's approach is better.
As someone who currently mostly programs in C++, lack of memory safety barely makes my top three concerns. My #1 problem with C++ is that the language is far too complex for (my) comfort, where by "complex" I mean requires too much effort to read and write. That, and build times, have a bigger impact on the correctness of the programs I write than the lack of sound memory safety. Would I be happier if, for a similar cost, I could eliminate all memory safety errors? Sure, which is why, if C++ and Rust were the only low-level languages in existence, I'd rather people used Rust. But I would be happier still if I could solve the first two, and also get some better safety as a cherry-on-top. Memory safety is similarly not the main, and certainly not the only, reason I use languages that have a (tracing) GC when I use them.
> Notice that the working approaches have something important in common: a strong system that, given certain assumptions, guarantees the lack of memory safety problems.
That's a very good point and I'm not arguing against it. It's just that even if it's true -- and I'm more than willing to concede that it is -- it still doesn't answer the question, which is: what is the best approach to achieving a required level of correctness?
The "soundness" approach says, let's guarantee, with some caveats, certain technical correctness properties that we can guarantee at some reasonable cost. The problem is that that cost is still not zero, and my hypothesis is that it's not negligible. My personal perspective is that Rust might be sacrificing too much for that, but that's not even what has disappointed me with Rust the most. I think -- and I could be wrong -- that Rust sacrifices more than it has to just to achieve that soundness, by also paying for "zero-cost abstractions," which, for my taste, is repeating C++'s biggest mistake, namely sacrificing complexity for the appearance of high-level abstraction that may look convincing when you read the finished code (perhaps more convincing in Rust than in C++), but falls apart when you try to change it. Once you try to change the code you're faced with the reality that low-level languages have low abstraction; i.e. they expose their technical details, whether it's through code -- as in C and Zig -- or through the type system, as in Rust. Zig says, since the abstraction in low-level languages is low anyway (i.e. we cannot really hide technical details) there is little reason to pay in complexity for so-called zero-cost abstractions.
Language simplicity goes a long way, even as far as sound formal verification is concerned. For example, there are existing sound static analysis tools that can guarantee no UB for C -- but not the complete C++, AFAIK -- with relatively little effort. It's not yet clear to me whether Zig, with its comptime, is simple enough for that, though.
It is my great interest in software correctness, together with my personal aesthetic preferences, that has made me dislike language complexity so much and made me a believer in "when in doubt -- leave it out."
And in that respect, it's not really the kind of software anyone needs to aspire to; aspiring to write programs simple enough that Zig will do the job is much more palatable.
Of course, Zig has an unfair advantage here in that it is not production-ready yet, and so it's not really "out there," and doesn't have to carry the burden of any real software (there's very little software that Rust carries, but it's still much more than Zig). I admit that when Rust was at that state I had the same hopes for Rust as I do now for Zig, so Zig might yet disappoint.
The only static analysis tools that can guarantee anything about C are doomed to have plenty of false positives. They are much less used in practice than tools ASan & co which don't guarantee anything but have way fewer false positive, if any.
They have their use-case, but calling them “little effort” is disingenuous.
That's where the effort comes in. Those false positives are removed by adding annotations or changing code.
> They have their use-case, but calling them “little effort” is disingenuous.
I said they require relatively little effort, because it's definitely less effort than a rewrite in Rust. This isn't hypothetical. >1MLOC programs in the industry today are checked in this way. If you have an existing large C program and decide that you need your program to have no UB, those sound static analysis tools are the most cost-effective way of doing that today.
With the former you lose all guarantees and fall back to a safe/unsafe duality, with the latter you need to rethink how your code works to comply yo the analyzer's mindset: in the end it's pretty much like Rust, but in an ad-hoc way, much less ergonomic.
> I said they require relatively little effort, because it's definitely less effort than a rewrite in Rust.
This is a ridiculous way to save your argument. You are well aware that this is not what “relatively easy” means.
It is comparatively easier to deploy such tools than to rewrite a whole project in Rust, but the payoff is also lower in the long run (Rust offers more than zero-UB), so we might get to a point (when tooling[1] and hiring pool have reached a point where it becomes sustainable) where the latter option makes more sense for most people (unless C is mandatory, for portability reasons for instance).
[1]: I'm especially thinking about C2Rust here https://immunant.com/blog/2020/01/quake3/
This is simply not true. The annotations are checked. It's exactly like adding type annotations when inference fails.
> with the latter you need to rethink how your code works to comply yo the analyzer's mindset
No. I'm talking about adding something like a bounds check in a function entry.
> in the end it's pretty much like Rust, but in an ad-hoc way, much less ergonomic.
Except that it is cheaper than a rewrite in Rust, which is one of the several reasons why this is currently the preferred approach in industry segments that require certain correctness guarantees. I don't know if you know this, but Rust isn't exactly making big headways in the safety/security-critical software world, especially, though not only, in embedded (for a multitude of reasons). Those sound static analysis tools, on the other hand, are showing nice growth.
Also, even if Rust were more ergonomic than this, there are alternatives that I think will be more ergonomic than Rust. I.e. it's not enough to be better than C++; if you want to get the people currently using C/C++ you need to be better than C/C++ in a way that justifies the transition cost and better than the other alternatives.
> You are well aware that this is not what “relatively easy” means.
Relatively easy means easier than all or most other available options. Anyway, that's what I meant.
> but the payoff is also lower in the long run (Rust offers more than zero-UB)
Nobody knows about the long term payoff Rust gives you because few people have had sufficient long term experience with it. It could be large, small, nil, or negative. And there are other languages as well. Zig, when it's available, might well have a bigger payoff than Rust (that's my current guess), and in any event, few people in the low-level programming space who aren't currently using C++ are even thinking about, let alone considering, Rust. Not that anyone is thinking about Zig, but at least Zig is aiming at C shops as well.
Again, as someone who has been using formal methods for some years now, I can tell you that nothing is obvious in software correctness, and no one knows what the best way to achieve it is (although we do know some best practices, and we have some answers to more specific questions).
> so we might get to a point (when tooling[1] and hiring pool have reached a point where it becomes sustainable) where the latter option makes more sense for most people
You are assuming that Rust is the preferrable choice. I no longer think it will be. BTW, here's "C2Zig": https://youtu.be/wM8vz_UPTE0
Rust is way too new for that obviously. And as I said, because tooling and hiring pool are not here yet, it would be a critical mistake to attempt such move at the moment.
For new projects however, Rust is a really interesting bet. (I was until recently working on a new medical robot, whose software was mostly Rust and the speed at which we got it working was really exciting!).
> in any event, few people in the low-level programming space who aren't currently using C++ are even thinking about, let alone considering, Rust.
That's not my experience. Not many are considering it for a lot of reasons (too new, not enough people mastering it, resistance to change etc.) But “thinking about” is another story ;).
> You are assuming that Rust is the preferrable choice
Zig isn't really a choice at this point. It might become it in a few years, but there still a long way to go.
The argument here seems to be that there is can be no real abstraction in low-level languages, so there's no point providing language features for abstraction. The premise seems clearly false to me, because even C has plenty of abstraction. Functions are abstractions. Private symbols are abstractions. Even local variables are abstractions (over the stack vs. registers).
People often argue that Rust is too complicated for its goal of memory safety. It's easy to say that, but it's a lot harder to list specific features that Rust has that shouldn't be there. In fact, as far as I'm concerned Rust is an exercise in minimal language design, as the development of Rust from 0.6-1.0 makes clear (features were being thrown out left and right). Most of the features that look like they're there solely to support "zero-cost abstractions"—traits, for example—are really needed to achieve memory safety too. For instance, Deref is central to the concept of smart pointers, and, without smart pointers, users would have to manually write Arc/Rc/Box in unsafe code every time they wanted to heap-allocate something.
> Language simplicity goes a long way, even as far as sound formal verification is concerned. For example, there are existing sound static analysis tools that can guarantee no UB for C -- but not the complete C++, AFAIK -- with relatively little effort. It's not yet clear to me whether Zig, with its comptime, is simple enough for that, though.
The most important static analyzers used in industry today are Clang's sanitizers, which work on both C and C++. The most important such sanitizers actually work at the LLVM level, which means they work on Rust as well [1]! The days of having to write a compiler frontend for static analysis are long gone. We have excellent shared compiler infrastructure that makes it easy to write instrumentation that targets many low-level languages at once. (Even in the world of C, this is necessary. Plain old C99 is an increasingly marginal language, because the real important code, such as Windows and Linux kernels, are written in compiler-specific dialects of C, which means that a static analysis tool that isn't integrated with some popular compiler infrastructure will have limited usefulness anyway.)
> It is my great interest in software correctness, together with my personal aesthetic preferences, that has made me dislike language complexity so much and made me a believer in "when in doubt -- leave it out."
Again: easy to say, harder to specify specific Rust features you think should be removed.
My argument is that low-level languages allow for low abstraction, i.e. there's little that they can abstract over, where by abstraction I mean hide internal implementation details in a way that when they change the consumer of the construct, or "abstraction", does not need to change; if it does, then the construct is not an abstraction. With "zero-cost abstraction," C++/Rust offer constructs that syntactically appear as if they were abstractions (e.g. static vs dynamic dispatch; subroutine vs. coroutine call), but in reality aren't. I am not aware of any other language (unless Ada has changed considerably since I last used it in the early '00s) that values this idea to such a great extent.
The things you mentioned are abstractions only in the sense that the user doesn't need to know how the compiler implements them; in that respect, every language construct, including `if` (e.g. in Java, not every if is compiled into a branch) is an abstraction. I speak of the language's ability to allow users to abstract, and in that regard all low-level languages provide for poor abstraction. Without a JIT, the caller needs to know the calling convention; without a tracing GC the caller needs to know how the memory pointed to by a returned value is to be deallocated. The question is how much you try to make sure that all this knowledge is implicit in the syntax.
> People often argue that Rust is too complicated for its goal of memory safety.
I didn't know people often say that. I said it, and I'm not at all sure that's the case. I think that Rust pays far too heavy a price in complexity. It's too heavy for my taste whether or not it's all necessary for sound memory safety, but if it isn't, all the more the shame.
> Most of the features that look like they're there solely to support "zero-cost abstractions"—traits, for example—are really needed to achieve memory safety too.
OK, so I'll take your word for it and not say that again.
> The most important static analyzers used in industry today are Clang's sanitizers
I'm talking about sound static analysis tools, like Trust-in-Soft, that can guarantee no UB in C code. I think that particular tool might support some subset of C++, but not all of it. The sanitizers you mention rely on concrete interpretation (aka "dynamic") and are, therefore, usually unsound. Sound static analysis requires abstract interpretation, of which type checking and type inference are special cases. Just as you can't make all of Rust's guarantees by running Rust's type-checker on LLVM bytecode, so too you cannot run today's most powerful sound static analysis tools -- that are already strong enough to absolutely guarantee no UB in C st little cost -- on LLVM bytecode; they require a higher-level language. Don't know about tomorrow's tools.
> Again: easy to say, harder to specify specific Rust features you think should be removed.
I accept your claim. In general, I don't like to isolate language features; it's the gestalt that matters, and it's possible that once Rust committed to sound memory safety everything else followed. But let me just ask: are macros absolutely essential?
Yes, for two main reasons: (1) type-safe printf; (2) #[derive]. Nothing is "absolutely essential" in any Turing-complete language, of course, but printf comes pretty close. The only real alternative would have been to use the builder pattern for formatting like C++ does (i.e. the << operator), but nobody seriously proposed that as the aesthetics are really bad and setting formatting flags like precision is problematic. And without custom #[derive], you couldn't serialize types, which is pretty important in a modern language.
Note that these are both cases (maybe the primary cases) in which OCaml has built-in ad-hoc solutions that feel very un-"systemsy". In OCaml, format strings are built in to the language, as are functions like PartialEq and Debug, the latter of which are implemented via magic built-in functions that call private internal reflection APIs. There was a desire to do better in Rust, as it was felt that these are the ugliest parts of OCaml, and so macros were part of Rust from the very early days.
BTW, Zig manages to do both things without macros and without any special cases in the compiler. TBF, Zig's approach was unfamiliar to me until I saw it in Zig, and it is Zig's "brilliant idea" (even if it had originated elsewhere), so it's OK if Rust simply didn't consider it; Rust certainly has its own brilliant idea.
Isn't having a UB-free JVM a noteworthy goal though? Especially if it gets used in life-critical systems such as avionics or autonomous cars.
> We always make some compromises on soundness; the question is where the sweet-spots are.
Excellent point. There are complex tradeoffs and the "rust is safe" slogan is just a slogan.
What kind of bugs do your have in mind that Zig would prevent and Rust doesn't ? I can think about several Rust does and Zig doesn't thanks to its afine type system, but I've yet to see a safety feature benefit in Zig.
If sound elimination of bugs were the best way to write programs, we'd all be writing in Isabelle, Coq or Idris, except even those of us who do -- in fact, especially those of us who do -- know that it's not the best way to write programs.
It might be possible. Someone might prove that preventing UAF necessarily increases complexity somewhere else. Like one of those laws of thermodynamics, but for software.
For example, I wrote a PEG-like parser combinator library in Zig. Using it currently [looks like this](https://github.com/CurtisFenner/zsmol/blob/87de4c77dd8543011...). However, as a library, I ^could provide a function that looks like
pub const UnionDefinition = comb.fromLiteral(
\\ _: KeyUnion
\\ union_name: TypeIden
\\ generics: Generics?
\\ implements: Implements?
\\ _: PuncCurlyOpen
\\ fields: Field*
\\ members: FunctionDef*
\\ _: PuncCurlyClose
);[^]: Being able to create brand new types at `comptime` isn't [yet implemented](https://github.com/ziglang/zig/issues/383), so this can't quite be done yet, though you could fake it with `get`/`set` methods instead of real fields
For many domains, being able to implement a domain-specific language with a set of expressive rules for the domain is an incredible productivity boost and also prevents many mistakes because you cannot represent them.
Not being able to manipulate the AST means that you are restricted on the embedded DSLs that you can provide. And embedded DSLs encompasses code generation for:
- state machines
- parsing grammars (PEGs for example)
- shader languages
- numerical computing (i.e. having more math like syntax to manipulate indices)
- deep learning (neural network DSL)
- HTML templates / generators
- monad composition
There is a reason most people are not building in Assembly anymore, there is a right-level of abstraction for every domain. A language that provides building block for domain expert to build up to the right level of abstraction is very valuable.
Low-level languages (aka "systems programming" languages) already suffer from various constraints that increase their accidental complexity. Is it really necessary to complicate those particular languages further to support embedded DSLs?
I don't think there's a universal right or wrong answer here, but there is certainly a big question.
It's very easy to come up with examples of terrible abuse of these features that lead to bad code (like for instance if somebody was insane enough to overload the binary shift operator << to, I don't know, write to an output or something) but it also gives a lot of power to write concise and expressive code.
That being said it does make it even weirder to allow braceless ifs IMO, but that's bikeshedding.
Which ones exactly? Not really a problem I've encountered often except when someone tries to mix both spaces and tabs, and in general editors are built with the existence of this very common character in mind. People tend to hit the tab key to indent anyway, and one tab char meaning one level of indent is perfectly intuitive and allows users to individually configure how large they want their indents to be.
for example ('-' is tab, '.' is space):
--function_with_lots_of_arguments(arg1, arg2, arg3,
--................................arg4, arg5, arg6);
of course, another style would be to just indent the continuation twice, without aligning it. (i personally prefer to align continuations.)
For example, this is the best way to define functions with argument lists long enough not to fit on a single line:
fn doThing(
argument1,
argument2,
argument3,
) {
<code>
}
More food for thought:
The style in your example is more consistent with the way nearly every programmer formats code that has more than one statement. For example:
if( shouldDoThings ) {
doOneThing();
doAnotherThing();
doLastThing();
}
if (shouldDoThings) {doOneThing();
doAnotherThing();
doLastThing();}
Somewhat related, the Rust coding style guidelines used to follow a heavily column-aligned style, but changed to indentation-only a year or two ago. I posted an example from the Servo source code with the old and new formats here:
>Not really a problem I've encountered often except when someone tries to mix both spaces and tabs
Mixing spaces and tab for indentation and alignment is how it should be done if you want things to remain aligned when you change the tab width.
Either way, it's a trivial algorithm to implement. Calculate len("function_with_lots_of_arguments("), add that many spaces after your indentation tabs on the next line, continue arguments. The fact that editors/auto-formatters don't do that speaks far more to their lack of desire for that code style then it does to the difficulty of doing it.
The simple fact remains that using tabs provides many accessibility benefits that spaces are just unable to provide, like being able to drop tab size when you boost font size (legibility for poor vision) so your code doesn't indent off the side of the screen, or far better support for indentation for proportional fonts (another thing that people swear by for increasing legibility).
With that out of the way...
>Either way, it's a trivial algorithm to implement. Calculate len("function_with_lots_of_arguments(")
That breaks for `foo(bar(a,\nb,\nc))`
And if you fix it to be "len till the last unmatched (" then it breaks for `foo(bar(a,\nb),\nc)` because the two lines have to indent to different levels.
Just saying it's not as trivial as it seems.
>The fact that editors/auto-formatters don't do that speaks far more to their lack of desire for that code style then it does to the difficulty of doing it.
And is a reason to not use such a style in the first place, as I said.
That's still just an argument for "everyone use a specific formatter with these settings". Go's formatting is simple, built into the main implementation, and has no knobs, as a result, everyone uses it, and all code is formatted with tabs and spaces. It can work, esp. if a language adopts it early (like Zig could have).
How it should be done is to never write code that needs alignment if you're using tabs for indentation.
It has worked just fine for decades for projects with millions of lines of code.
It might not work for babby's first patch and how do I configure an IDE??
It's simply not worth it IMO. The pros are simply too small and inconsequential to justify not using spaces on new projects. Although now that automatic code formaters are becoming ubiquitous it really doesn't matter what you use in your editor I suppose.
Throw in their plans to add a package manager and it's own make system and it seems to be getting further and further from the C replacement I was interested in.
I have seen both Zig and Swift use the module system very well despite several devs strangely saying its somewhat complicated, but when used for low-level development this makes Zig an interesting language to use for language bindings.
Already importing blas into elixir with it is crazy easy.
The problem is: do you want to maintain full backcompat or are you willing to break things to do it cleanly? If you do the former you end up with something like C++ which retains almost complete compatibility with C (but they you have a lot of baggage to carry around, which may get in the way at times) or you're willing to break things and then why not take the opportunity to improve the syntax and get rid of the cruft?
C is ancient now, its type system and its many quirks are quite far from the state of the art or language design, a modern language wouldn't have to mess with the nonsense that are, for instance, C arrays (including C++ which can't outright remove them but does everything it can to render them obsolete with std::vector and std::array).
An other big problem with C interop for newer languages is that while C itself is relatively small and easy the C preprocessor isn't. That's usually where the friction is, because if you want to maintain compatibility with C macros you have to choice but to implement the language syntax wholesale.
In retrospective, the aversion of Go designers to current practices in other programming languages is quite similar to the aversion of them back when designing C, versus the other system programming languages that were being developed since 1961, like ESPOL, NEWP, PL/I, PL/S, BLISS.
That's a pretty awesome place to be, as C++ is one of the most successful programming languages in history.
Perhaps there are lessons there.
> or you're willing to break things and then why not take the opportunity to improve the syntax and get rid of the cruft?
Break things just for the hell of it is not much of a tech argument.
C is already pretty light, and already gave origin to successful programming languages such as C++ and Objective C. Unless you find a compelling reason to break backward compatibility in a very specific way, I don't see how that argument makes any sense.
> An other big problem with C interop for newer languages is that while C itself is relatively small and easy the C preprocessor isn't.
Arguably, the preprocessor is orthogonal to the programming language itself. I fail to see how that's relevant.
C is light but it does have some things worth breaking IMO. Type inference is something I dearly miss when I write C these days (and I do that a lot). C didn't have any generic programing for a long time (if you don't count macro soup, that is), now it has some very limited support but it still looks like banging rocks together compared to more modern languages.
C's unsafety is legendary, and segfaults a common problem even for experienced programmers. Rust's lifetimes makes them impossible by design for safe code.
You may not like that of course, but those are all good reasons for experimenting with other paradigms.
>Arguably, the preprocessor is orthogonal to the programming language itself. I fail to see how that's relevant.
Arguably it is, practically it very much isn't.
Note that Objective-C and C++ are extensions to the C language, and both started as pre-processors that would generate C code.
Also C wasn't the first on its domain, just got lucky that UNIX got widespread adoption and then found its way outside UNIX, just like JavaScript eventually found a way outside the browser.
Because extending or improving a language nearly always means breaking compatibility. Nearly every language is "stuck in a local optimum". To get out of it, you have to kill assumptions of the language that lead to this local optimum.
Also it takes a lot more work to convince other people to adapt my changes than to write an implementation of a language. Add to this the fact that being a good programmer and being a good politician are rather independent skills.
Finally, there do exist issues that cannot be fixed, for example that the official ISO standard of the C language is not freely available (I am aware that there exist drafts in the internet).
Compare this to merely extending a language e.g adding smart pointers - you could always (mistakenly) implicitly fall back onto bad coding practices.
https://www.youtube.com/watch?v=Gv2I7qTux7g
(the title & abstract in the youtube description is the one I gave for the RFP; I ended up going in a slightly different direction than it once I actually made the talk)
C but with the problems fixed.
I am all for any language that helps reduce C's usage, including Go, even when I dislike its design decisions.
Now Zig, if it doesn't fix what are about 60% of yearly expenses fixing security exploits, according to Google and Microsoft security reports, then one needs to analyse how much is it worth actually.
According to this resource, Zig produces code that is very close to the hand written assembly for the simple use case of outputing "hello world" on the stdout.
This is to be taken with a grain of salt though, as of course, caring about the assembly output is a spectacular case of premature optimization. I guess it tells a bit about the goal of Zig as a low-level programming language though.
I don't agree! at least for operating systems.
Consider that some operating system code may be run a million times per second, on a million different machines (e.g: block system IO on linux). We very much want our assembly to be pristine in this case.
I also like the idea of "optimality" brought forward by zig. In the post you link, there's an ideal hello_world in asm, can we have a higher level language that doesn't sacrifice this ideal achieved by assembly?
Consider that some network cards are now capable of 400Gibps, and modern OSes are not capable of handling these linerate. I strongly believe the bottleneck should be in the hardware, if the software can't max out your hardware then your software has failed.
I like that zig focuses on optimality.
I would believe that most of the time of a processor is dedicated to run userspace code, not OS code (regarding to the work that must be done). At the end of the day, the OS is "only" a scheduler to share hardware resources between unrelated tasks.
> I like that zig focuses on optimality.
Optimality isn't required in many real world businesses. Optimality is often a tradeoff : with optimality you lose in flexibility. An optimal program with SISD instruction set isn't optimal anymore when SIMD instructions are introduced. Your "optimal" asm program is still optimal on x86, but also totally obsolete because it cannot use the latest NEON instructions from ARM or 64bit instructions.
Yeah, that's true if the OS is well-written and has had continual and extensive performance work done. Passing 200 Gbit of network traffic in software with firewalling is non-trivial and the number of cycles you get per packet is pretty modest. These things do matter.
If that's the only function of the program, are we still talking about an "Operating System" ?
This is a hobgoblin. Assembly output of a compiler is the very baseline of performance. It requires no effort on behalf of the programmer (aside from learning a language).
Now, if you're skimming the assembly after every compilation and, say, fuzzing your implementation to coax the compiler to emit the best possible assembly... that's probably premature. If you're writing inline assembly before doing a higher level implementation, that's probably premature.
But choosing a language on the basis of its performance:effort ratio is downright pragmatic.
Can be important for kernel stuff.