Build your own SQLite in Rust, Part 5: Evaluating queries
blog.sylver.dev
blog.sylver.dev
So, which of these points apply?
I think it means you might use it in similar situations. Like when you want an in-memory database.
Read my post again. Sympathy or lack thereof don't factor into this equation. This is about battle-testedness and features. I don't make technical decisions based on emotions.
- Limbo: "Limbo is a work-in-progress..."
- Sled [1]. Not sure what's going on there. Last release 3 years ago, but a constant stream of "alpha" versions that never get released.
SQLite with Rust bindings seems to be the go-to system. Depending on C packages is often a headache when cross-compiling, though.
[1] https://github.com/surrealdb/surrealdb?tab=readme-ov-file
Either you need to make it easy and zero friction to adopt like duckDB and let people find out themselves in an hour or two or you need to provide some sort of benchmarks + evidence that it isn't going to die on its arse the moment you put larger than memory amounts of data in.
Nearly all of these projects work fine for in memory size datasets but only finding out after you've put major effort into adoptionv+integration isn't really easy for someone working with data when you have something battle tested like Postgres et Al.
The result is that most Rust crates sit on the 0.x version forever (since anything before 1.0 in semver is pretty much a free-for-all), even though they're probably going to be perfectly usable in most cases with the same library breakage you expect in other languages.
IMO it just shows another dent in thinking that a simple string can confer any information about the API contracts on offer. Semver is a neat indicator, but practically that's all it is.
But the years of work put into the existing project to make it robust don't mean the exact same years have to be spent on the reimplementation:
- there's been work spent on discovering the right architecture and evolving the db format. A new impl can copy the end result.
- hard lessons have been learned about dealing with bad disks, filesystems, fsync, flaky locks, etc. A new impl can learn from the solutions without having to rediscover them the hard way.
- C projects spend some time on compatibility with C compilers, OSes, and tweaking build scripts, which are relatively easy in Rust.
Testing will need a clever solution. Maybe they'll buy access to the official test suite? Maybe they'll use the original SQLite to fuzz and compare results?
Before, I have the myth in my head that RDBMS development is like cryptography/kernels: Something not mean for humans to do.
Now working as part of one (http://spacetimedb.com) I see now that is hard, but doable.
Certainly MORE EASY than working in ERPs that is the thing that I have done for more than 20 years (now THAT is what is insane). Literally doing RDBMS is more relaxing than doing ERPs.
But what I have learned is that what make RDBMS truly hard is not the usual suspect: ACID.
You don't need 'years of hardwork' to reach it and to prove them is right (only some insanity when you need to target very weird os/archs and deal with faulty hardware).
What is truly hard and will eat your time is sql. SQL is hard. Is *bad*. Is *very hard* to optimize, and is so anemic that your time is expended more in the query optimizer (that has hard time because sql more than anything else) and because sql is bad, I tell you, it leads to insane stuff like having a single query with 1000 joins, and people have not means to good, optimal designs (and what you have like CTE and all that is poor bandaids) so now you get a bad input and have fun.
But if the RDBMS stay in the relational part and you could make a better lang to interface it then is far more easy. Still query optimizer is big part of your time, but if you have control of the input language you can do your life easy.
I completely get why SQL is important and as an user I love it as a language, but as an application writer I have a very complicated relationship with it. It requires either inline SQL (which carries its own security risks and causes a bit of redundancy), or we gotta use complex abstractions on top of it, like ORMs. Tests also require spinning up a database or require some extra machinery.
It feels like we're making our lives more complicated by "requiring" a database to have SQL.
I would be totally ok with adopting a non-SQL relational database with a more structured API in greenfield projects. (Btw I will be definitely checking out your company).
It's up to the developers to decide if the outcome is worth the effort.
Maybe they're right, maybe they're wrong, but either way – if they succeed, we're all better off for it, aren't we?