How I Built Zig-SQLite
rischmann.fr
rischmann.fr
(A bit off topic rant, but Zig documentation is quite bad, it took me a lot more effort that it should to discover the facts above.)
[1] https://ziglang.org/documentation/master/#Anonymous-List-Lit...
[2] https://ziglang.org/documentation/master/#Anonymous-Struct-L...
I used "ziglearn" to understand ("chapter 1" and "chapter 2" have tons of foundational stuff): https://ziglearn.org/chapter-1/
I used the language reference for the details: https://ziglang.org/documentation/master/#Introduction
I agree the standard library docs aren't useful (unless something changed a lot in the last six months)... it seemed always better to just search and read the standard library source directly.
They're automatically generated. As I understand it they've been holding off on improving them until they have the self hosted compiler working, as it will be easier at that point.
The dot seems quite superfluous and quite the silly quirk (in an unintuitive way). How do Python and Go get by without such a mystery sigil?
SELECT * FROM user WHERE age = $age{u16}
For what it's worth there are in zig-sqlite variants of the method which bypass the comptime checks; they're not documented properly yet but see all methods named `xyzDynamic`, for example https://github.com/vrischmann/zig-sqlite/blob/master/sqlite....
Can you call comptime-intended code at runtime? No? (Yes? B/c the call site is "in" the runtime code?) But just make it runtime code instead of comptime code?
How's the Language Server support for this? Last time I tried ZLS, it was quite well rounded so I am curious to know if this type annotations would work with it or not. Would be really cool if they do.
I have never been able to fully adopt zig due to how frustrating working with strings is but I absolutely loved the comptime functionality.
It would be cool to have an official Zig language server that can do it
For example, from the article "Why LSP?" [1]
"It is known that compilers are complicated, and a language server is a compiler and then some."
* arbitrary file I/O. I can take my compile time data and write a python script to put it in a std::array, but I shouldn’t have to.
* non-fixed-sized containers, ie vector
* a generic memoization utility in the standard library for caching. If I want my function to handle any input at runtime, but I want to pre-populate a cache at compile time for values that I know will be called, it’s doable but not as easy as it should be. (In general, given the amount of algorithms that rely on memoization, I’m somewhat surprised that Python is the only language I know that makes memoization as easy as a decorator).
https://ziglang.org/documentation/0.9.1/#embedFile
I haven't tried, but because it should just be equivalent to a string literal you should be able to further process it using comptime as well.
> * non-fixed-sized containers, ie vector
One way is to just have a function count how many entries you need and use arrays. This is all pretty straightforward because comptime.
Real dynamic comptime containers are dependent on making a comptime allocator available. There's a ticket for that: https://github.com/ziglang/zig/issues/1291
> * a generic memoization utility in the standard library for caching. If I want my function to handle any input at runtime, but I want to pre-populate a cache at compile time for values that I know will be called, it’s doable but not as easy as it should be. (In general, given the amount of algorithms that rely on memoization, I’m somewhat surprised that Python is the only language I know that makes memoization as easy as a decorator).
I don't think it's possible to use comptime to generate wrapper functions duplicating the signature of a given function, at least I haven't found a way to do so. The function signature has to be spelled out in the comptime code.
on the same note, lets say if someone is creating something[i am thinking of] like this, what would be a better option, go the forth way or the lisp way cuz both offer same stuff to some extent
i think ive realised what is up with lisps, forths, etc. theoretically you can build all kinds of beautiful abstractions over it and use them for anything
but in reality these languages cant do that and/or their communities dont care to implement it/its not their goal
it might sound bad but it is what it is
if i would have to say it in a worse way, these languages are all talk
I can give you the Rust perspective, but I'm not sure it's the best.
> arbitrary file I/O
include_str! and include_bytes! make the contents of a file available as a string or a byte array. More complex types would need a build script (or transmute).
> non-fixed-sized containers, ie vector
No support in `const fn`, but you can again use a build script as an escape hatch.
> a generic memoization utility in the standard library for caching
There's nothing in the stdlib, but there are a few third party crates that give you the easy one-line syntax, such as https://crates.io/crates/memoize
There's a proposal for that https://open-std.org/JTC1/SC22/WG21/docs/papers/2020/p1040r6...
> non-fixed-sized containers, ie vector
You can already use them in a constexpr context, you just can't leave it yet. So creating a vector on compile time and then using it on runtime is sadly not working right now but I think that's being worked on as well.
SQLite already compiles queries into opcodes and then uses its own VM to interpret them. You would have to reimplement the internal VM.
No, it doesn't. C does not even assume a stack. A platform where function call frames are allocated on the heap would not be incompatible with C.
You could say that some C code assumes a stack, but that's pretty exceptional.
> Although I imagine that currently, in most cases, you'll want to avoid C for software reasons,
Yes.
> rather than hardware reasons
Hard to imagine hardware on which C could not run. Maybe a JVM chip, but even then, you could compile C to bytecode.
The problem is not with call frames being on the stack or on the heap; even a spaghetti stack would be problematic. One of the problems is that return addresses and data are interleaved in C-style frames, whereas multiple stack machines require them to be separate. It's not that you couldn't write an implementation of C for a stack machine, it's just that it would be probably very primitive and slow. Yes, you can run C at the very least the way it was done on Lisp machines, by allocating a large byte array and treating it as your physical memory, but surely you would want to avoid it if you could. There really is a reason why stack machines historically ran somewhat exotic languages like Forth.
HLSL and CUDA have C-like syntax, but underneath the syntax these things are very different from C.
On GPU almost nothing has an address, there's no stack, no malloc/free, no files or printf, and every instruction runs on 32+ threads in lockstep.
And here you're proposing Zig while someone else will prefer Rust, and someone else Go. So you'll need N>1 rewrites.
This is why SQLite3 is written in C then, because for all the ways in which C sucks, it is the most common (lowest) denominator that ticks the portability checkbox.
Language runtimes that want to not have a C runtime embedded are the hardest hit. The best thing to do for those in the short-term is to talk to a C-coded IPC service that runs the C-coded thing that otherwise can't be run.
This is actually something that Zig is supposed to do as well. There should be nothing that you can do in C but can't do in Zig. So hypothetically, in the future (say, twenty years from now), a Zig implementation of SQLite could be a preferred one since you're not losing anything with it. (With the additional benefit that should you want to, you might be able to compile application-specific queries into it.)
Great work of the author explaining comptime!
I can't figure out why i am supposed to care about zig beyond it's fun? I get that it's interesting and more safe than C (honestly though what the hell isn 't).
Say you write rust pretty regularly for new product development, what does zig offer to make my life better, my products more stable, etc?
In all honesty, I prefer the syntax of D over all the others, it feels the most like Java or C# but with a lot of modern benefits. D is trying to do too much though it feels like and I would love for the next D standard library to support OOTB similar to what Go supports, especially a very minimalist web server, I think all modern programming languages should be capable of spinning up web servers out of the box. This is one small detail Go got right in my opinion.
Here's an article from the Chromium team on challenges they faced with trying to integrate Rust (or at least evaluating it) note the entry was last updated in 2020:
https://www.chromium.org/Home/chromium-security/memory-safet...
(To be fair, I think Python and Ruby have HTTP servers in their standard libraries as well, at least minimalist ones.)
At the other extreme, if you're writing an operating system or a language VM, you probably want contextual allocators (like an allocator that takes a runtime argument like "which green thread I'm allocating on") which rust makes extremely difficult.
If you really need memory and resource safety I think the best answer is to be patient. Zig is very easy to parse and I imagine it will be the case that static analysis build tools will come about which can do what you want out of rust.... Being decoupled from the compiler chain you would be able to run fast but guard your prs to main/dev/release (as you see fit) with static analysis tools that will protect you with the safety you seek in an isomorphic fashion to "how rust does it". There's no reason why someone couldn't write it now, but with a lot of things (like ZIR/AIR) being highly unstable -- and these are what you're likely to want to statically analyze for such a tool -- for ones sanity I don't recommend building out a tool like that now.
Also it's not like that safety costs much of anything. You really can get comfortable writing rust, especially "easy rust" like a single thread cli app. It really is easy, even convenient.
Either zig is a different tool to rust, in which case I get it. Or people should be waiting for zig to be mature, meanwhile rust has been stable for half a decade, and at this point "just works". I'm not going to sit through an addendum static analyzer and all the bugs that come with making one for a few years. Other people whose appetite for risk is higher will though, and I wish them the best. I've seen efforts in other languages, my conclusion is that it's much better to have it built in...
==========
Why Zig When There is Already C++, D, and Rust?
- No hidden control flow. If Zig code doesn’t look like it’s jumping away to call a function, then it isn’t.
- No hidden allocations. Zig has a hands-off approach when it comes to heap allocation. There is no new keyword or any other language feature that uses a heap allocator. The entire concept of the heap is managed by library and application code, not by the language.
- First-class support for no standard library. Zig has an entirely optional standard library. Each std lib API only gets compiled into your program if you use it. Zig has equal support for either linking against libc or not linking against it. Zig is friendly to bare-metal and high-performance development.
- A Portable Language for Libraries. Zig is attempting to become the new portable language for libraries by simultaneously making it straightforward to conform to the C ABI for external functions, and introducing safety and language design that prevents common bugs within the implementations.
- A Package Manager and Build System for Existing Projects. Not only can you write Zig code instead of C or C++ code, but you can use Zig as a replacement for autotools, cmake, make, scons, ninja, etc. And on top of this, it (will) provide a package manager for native dependencies. This build system is intended to be appropriate even if the entirety of a project’s codebase is in C or C++.
- Simplicity. Zig has no macros and no metaprogramming, yet is still powerful enough to express complex programs in a clear, non-repetitive way. Even Rust has macros with special cases like format!, which is implemented in the compiler itself. Meanwhile in Zig, the equivalent function is implemented in the standard library with no special case code in the compiler.
- Tooling. Zig provides binary archives for Linux, Windows, macOS and FreeBSD. It is installed by downloading and extracting a single archive, no system configuration needed. It is statically compiled, uses LLVM, has out of the box cross-compilation to most major platforms, and ships w/ libc source and dynamically compiles when needed. The Zig build system has caching and compiles C and C++ code with libc support
==========
To me it comes down to Zig is a "modern" C, but unlike most other attempts at replacing C it doesn't skip some of C's use cases.
This one is an odd point; this very article is a demonstration of metaprogramming*:
> Thanks to Zig’s type reflection we can read a row of data into a user-provided type without needing to write any “mapping” function: we know the type we want to read (here the User struct) and can analyse it at compile-time.
* Which is a feature I favor, for the record.
- I use zig as my build system for both rust, zig, C libraries and linking since the build system works really well for this purpose
- When I need to write applications or libraries that can benefit from compile-time code, I always try and use zigs since it's much easier to use comptime then a combination of rust macros and generics
- I like the zig async story a lot better. Or at least it's much easier to wrap my head around and write code in compared to rust + tokyo
On the other hand, sometimes I know a project will benefit from the borrow checker or I want to use some of the awesome rust crates that the community made and I'll use rust instead.
Agree, Tokio is a little tricky for people new to rust no doubt, but it's tricky for a reason. It's saving lives in production.
(The way to manage memory in C++ and C is to opt in to a memory management pattern... hopefully you've chosen a good pattern and hopefully you follow it consistently. Patterns are generally backed by utilities and primitives that are hopefully correct, complete, and hopefully make it relatively easy to follow the pattern consistently. Hopefully you can get a linter to help you too. In a sense Rust takes the same approach, but was implemented from the ground up to eliminate all the gaps so you essentially don't have to hope to successfully bridge the gaps yourself.)
While you can technically use Rust in these domains you'll find yourself jumping through hoops and fighting against quirks that come with it being fairly high-level and very opinionated on how to enforce memory safety.
One such scenario I've encountered is implementing my own memcpy with a loop like `for i in 0..len`. This works in release builds, but without optimisations this gets a deeeeep callstack that eventually also calls memcpy, so you get a stack overflow. Note how memcpy is implemented in rlibc to avoid this issue: https://docs.rs/rlibc/latest/src/rlibc/lib.rs.html#30-38
It's funny how in rust you can write raw asm, but there seem to be quirks somewhere in the middle.
- The small footprint in Zig enables fast compilation (and getting faster) and fast feedback cycles.
- Arbitrary nastiness can happen using From and other traits in Rust, and the tendency to shadow names and rely on type inference can make that unpleasant to track down. In Zig you'd be forced to make a choice (for any non-trivial type) at the return or call site of how you were going to convert things.
- If you want to use the stdlib and have any non-trivial control over how objects are allocated you're in for a rough time in Rust (say, a circular buffer backing some objects and a reusable arena for others). You'll probably be reinventing a lot of wheels.
- Similarly with issues like implicit locking in stdout. It's executing a syscall per line anyway, but the lock can inadvertently make contended writes 1000x slower, so suddenly logging in multithreaded code needs a dedicated logger, and not any of the common options since those fall back to the locking stdout we're trying to avoid.
The language was not small whatsoever at the time. It was not fun.
It may not be the smallest language, but, it compiles pretty tightly.
Rust can provide some guarantees but it still won't protect you from logic errors or a "clever" coworker. In comparison, zig is simple and at the end of the day readability is my best defense against bugs/errors.
There was an interesting comparison (and discussion) of zig vs rust safety a few months ago: https://news.ycombinator.com/item?id=26537693
I find myself agreeing with pron and others in that discussion when they say no one really wants to use a safe language. What they want are correct programs.
What remains to be seen is how easy it is to write correct programs in each and the values people place on the deltas between the two languages.
This isn't really consistent with my experience. Compared to writing other languages with varying levels of strictness (C, Python, Go to name a few), Rust's compiler saves me a lot of time writing tests and finding bugs.
It doesn't save me the work of fixing them, but in my experience, bugs that are hard to find and easy to fix drastically outnumber bugs that are easy to find and hard to fix.
That said, I haven't tried Zig. It's on my radar, but just haven't had a new project to start lately.
Once I started realizing what the borrow checker was saying it wasn't "fighting the compiler" I realized what I was asking the compiler to do was really not a good idea. It becomes your friend basically.
I didn't make this post to shill rust, I'm pretty language agnostic to my own detriment... The point is, it's worth considering and trying it again if you didn't get it the first time around. Rust took me 2x to become a tool I'd use, I honestly hated it at first.
I like rust (I actually liked it from the start years ago). However, I believe that there are tradeoffs and the companies you named use other languages in addition to rust because they also believe there are tradeoffs.
Rust won't be the last language. I don't want to get too comfortable with what I know and miss out on what I don't. I am excited by new languages that may also have something to offer. However, I can't master them all so a good discussion is valuable and can offer shortcuts through the noise. I think my assumption in my first reply to your original post was that you were interested in the same thing. However, if you are happy with rust then I don't want to spoil your enjoyment.