Building a Database in the 2020s (2022)
me.0xffff.me
me.0xffff.me
We have a few very good memory safe programming languages at this point. Please do not start a project in C/C++ unless you are truly exceptional and understand memory management and exploitation inside and out. I switched to SQLite on the project since it is one of the more fuzzed applications out there that fit the need. The next embeddable database I use (bonus if it works on cloud) will need to be in a memory safe language.
I’ve been working on an iOS app lately with some of the core code in rust (shared with other platforms) and the UI in native swift. Changing the API at the boundary is a bit annoying, but the resulting app works great.
I'm using the swift_bridge crate to generate a lot of the bindings and generate the .xcframework.
https://github.com/josephg/replica/blob/master/build_swift.s...
Java's in a weird spot. On the one hand it almost definitely gets way more shit than it deserves. On the other hand there's a kernel of truth in a lot of the shit that comes its way especially in this area.
Low level programming in Java is unbelievably awkward. From 2Gb limits on bytebuffers to the absence of unsigned types. Everything is just a bit too far removed from the actual hardware and OS to do this stuff really well.
It's like coding with oven mitts, and undeniably there's a bit of overhead almost everywhere as a result, beyond what the JVM already introduces (which is not a lot).
Another big cause of awkwardness in this space is the lack of generic algorithms. Yeah you have <Generics> but they come at a cost of boxing all your primitives, which is just untenable if you need to go fast. So because you're undoubtedly dealing with some awkward custom workaround to the 2Gb byte buffer limit, you end up having to build and maintain a custom standard library of N different sort functions, N different binary searches, N different merge functions, N different bisect functions, N different B*-tree (or whatever) implementations, etc. to support different N widths of data types (probably at least short, int, long, float, double in a dbms).
Although it should be noted a lot of this is getting better with JEPs that are in the pipe and ought to arrive in a few years.
Similarly, it has good support for struct (value) types, and it boxes native types far less often than Java does, for massive performance benefits. Last but not least, NET 8 will come with an ahead-of-time native compiler for faster startup times. It's about to get AVX-512 support! [3]
[1] https://learn.microsoft.com/en-us/dotnet/api/system.runtime....
[2] https://learn.microsoft.com/en-us/dotnet/api/system.span-1
Storage people have a very strong allergy to anything that comes from Microsoft. You probably would have a better chance of writing storage applications in Common Lisp than C#. Only Microsoft uses its own storage products in places that matter. Even just having to integrate with Microsoft's products is seen as a huge pain that's not worth the effort.
And, truth be told... I want it to stay that way. Companies like Microsoft should be kept as far away as possible from critical infrastructure. We failed on few fronts, and are paying through the nose for it. It's nice that at least on this front, this is not a problem.
Also, on a personal level, no other language makes me want to puke as much as C# does. I mean, to the point that if the only programming jobs available in the world were the ones that ask for programming in C#, I'd go farming or cleaning... whatever. Just not touch that steaming pile ever again.
I've known about a bunch of "sister" storage projects which were written in a similar combo: typically the product itself would be C (or eventually, D, more recently -- Rust), but a language like Java would be inconceivable for a system like that.
Few things that these projects had in common:
* Home-made concurrency (i.e. they didn't use pthreads, instead they used some green threads designed specifically for this project).
* Home-made allocator. Again, similar to the threads, the default allocator options available to system programming aren't good for mission-critical software.
* Home-made logging. Due to high volume, speed and storage requirement it's typically a very involved system that is far more sophisticated than any Java built-in or 3rd party logging library.
* Home-made collections / containers. The reasons to have this is very similar to the reason to have custom allocators -- improved ability to deal with memory-related errors.
* Home-made drivers and IP stack, or at least partial re-implementation of IP and higher-level protocols. Again, because of memory-related concerns.
* Virtually no 3rd-party libraries. For close to 2M lines of C code for the core part of the system, we had like... some compression algorithm library as a dependency... and that was it, iirc.
I mean, these kinds of products need to rebuild something rather minimal like C almost from the ground up... just the mere thought of using Java in this kind of environment would've been laughed out of the room.
I know that databases, like relational databases are often written in Java, but in my experience, those are mostly used by Java applications and rarely leave the boundaries of the Java ecosystem. People coming from storage background rather than pure Java background wouldn't think about Java databases more than as toy projects really.
I am happy to use Java if I am doing graph database work or forced to interact with an Apache project. It is fine language and the developers are cheap. Often a good choice.
Do you have an example of something that you previously saw as a black box but now understand? Perhaps something simpler than you anticipated, or just something you never even knew existed before digging into the details?
But ok sure I can give you some pointers to cool stuff. Sometimes it are just architectural patterns that are easy to do like upsert statements in sql using `on conflict`, or writing a task queue on the cheap using `for update skip locked limit 1`. Replace 90% of your crud api controllers with `postgrest` and use row level security for everything, AI is not going to steal your job category theory is. From a operations and scaling perspective it pays dividend to learn about `explain analyse` but did you know they also have `stats` that can help the analyser optimise queries based on statistical relations of different columns. I've also been looking into how to "branch" a database instead if just doing a backup, it does require some ZFS tricks but that is also just pure power and not something you find with a cloud file system. There are also just a ton of extension for timeseries or vector embeddings, or write your own in Rust using `pgx`. Like I said way too much to write here, I don't use all of this in production but I just keep finding these gems while working on my own project. There is a strong OSS ecosystem with lots of teams giving there own spin on pg and that is welcomed if you have a opinion of your own and still want to learn from others. You need motivation to go this deep but in contrast to other esoteric knowledge I have mastered companies are also willing to pay for it.
The Apache webserver (and most other webservers) is another example like Postgres where it has a ton of useful features no one uses.
Some of these less popular features don't work at massive scale but in my experience they work great for smaller teams where adding a new tech to the stack is more painful.
In the cloud you can just snapshot/clone the underlying EBS volume and you’ll get a “branch” of any database on any file system, right?
If your want a full featured product (in the postgres space) that does it then look at neon.tech (no affiliation).
supabase is essentially a Postgres database with PostgREST on top, and we recommend pushing down a lot of the logic and security into the database. We took this philosophy with our pg_graphql extension (which uses pgx) and it is faster than other graphql implementations, simply it's co-located with your data, solving the n+1 problem.
pl_rust just reached 1.0, and it is now a "trusted language" so you can expect to see it arriving on a few cloud providers soon. We are releasing something this week with the RDS team which will make it easier to write key parts of your application code in trusted languages. There are certainly trade-offs, and I don't know if _everything_ should be in the database. But in data-intensive cases it makes a lot of sense.
- JWT is used to authenticate API requests. The JWT contains a `role` claim which is a PostgreSQL role that is then used for the duration of the request. This role is subject to regular PostgreSQL security, be it table, column or row-level security[1].
- You expose a subset of your database objects for your API schema. This schema consists of views and functions(or only functions) to hide internal details to API users[2].
[1]: https://postgrest.org/en/stable/auth.html#authentication-seq...
[2]: https://postgrest.org/en/stable/schema_structure.html#schema...
ZFS can also perform compression, which could be a big win as long as you have the spare cycles for lz4 (which is normally the case but may be an issue if you’re maxing out your NVMe disks’ bandwidth, say 3GiB/s+)
I read to the end of the article, and while having a database as a serverless collection of microservices deployed to a cloud provider might be useful, it ultimately will be useless if this swarm approach doesn't give me any guarantees about how or if my data actually makes it onto persistent storage at some point. I was expecting a discussion of the challenges and pitfalls involved in ensuring that a cloud of microservices can concurrently access a common data store (whether that's a physical disk on a server or a S3 bucket), without stomping on each other, but that seemed to be entirely missing from the post.
Performance and scalability are fine, but when it comes to databases, they're of secondary importance to ensuring that the developer has a good understanding of when and if their data has been safely stored.
There, you can $$$ your way out of data corruption. You can even loss all the data if you have enough replicas and backups.
Not many are in the game of Sqlite.
This is the space I wish to work more: I think not only mean you can do better the high-end but is more practical all around: If you commit to a DB that depends of running in the cloud (to mask is not that fast, to mask is not that reliable, for extract more $$$ from customers, mostly) then when you NEED to have a portion of that data locally, you are screwed and then, you use sqlite!
There, you can $$$ your way out of data corruption. You can even loss all the data if you have enough replicas and backups.
You can't buy your way out of having to deal with the fundamental problem of, "How do I provide the illusion of a single unified system for a highly distributed swarm of microservices?"
[1]: https://en.wikipedia.org/wiki/Fallacies_of_distributed_compu...
In the absolute case, yes. But large companies have lost a lot of data, and thanks of $$$ they survive that.
In that scenarios, you pay a lot to not get in problems, but you can pay to overcome them too ("pay bigly for lawyers to follow the law and not get in trouble, and to broke the law and get away with it").
Is not the ideal and exist a point in where that could break badly, but with certain size fatal software failures that will destroy a small company are just "Thursday" for somebody big.
BTW: I don't like this, I prefer to make software solid, but everybody run on C, Js, MongoDb, etc, and this show you can survive a massive crash...
As an industry, we've progressed quite a bit from accepting data isolation level compromises like "eventual consistency" in NoSQL, cloud, and serverless databases. The database I work with (Fauna) implements a distributed transaction engine inspired by the Calvin consensus protocol that guarantees strictly serializable writes over disparate globally deployed replicas. Both Calvin and Spanner implement such guarantees (in significantly different ways) but Fauna is more of a turn-key, low Ops service.
Again, to disclaim, I work for Fauna, but we've proven that you can accomplish this without having to worry about managing clusters, replication, partitioning strategies, etc. In today's serverless world, spending time managing database clusters manually involves a lot of undifferentiated, costly heavy lifting. YMMV.
Even if some aspect of Rust safety can't cover the complications of the low-level stuff, is easier to do that on Rust than with C/c++.
Generally speaking, modern database architectures have minimal dynamic memory allocation, no multi-threaded code to speak of, or even buffer overflow issues (everything internally is built for paged memory). Where are these advantages from Rust supposed to come from? It has a lot of disadvantages for databases today, and I say that as someone who has used Rust in this space. People who build this kind of software are pragmatic and Rust doesn’t deliver for database developers currently.
Two core design elements of modern database kernels create most of the real challenges. First, all references to most objects are implicitly mutable and some references are not observable by the compiler; safety can only be guaranteed dynamically at runtime by the scheduler. Major performance optimizations rely on the implications of this. Second, your runtime data structures don't have lifetimes in the ordinary sense because they all live in explicitly paged memory -- they aren't even guaranteed to have a memory address, never mind a stable one. And you want this to all be zero-copy, because performance. There are elegant ways of doing this transparently in C++ with a bit of low-level trickery, but it is antithetical to the way Rust wants you to manipulate memory. There are workaround options in Rust of course but they are strictly worse than just doing it in C++.
Database kernels are an edge case. Most systems software doesn't break most assumptions about ownership and lifetimes so pervasively. I expect Rust will add features over time that help in these cases.
OLAP has certainly become overcomplicated but collapsing everything into big monolith dbs again is an over correction IMO.
[1]: https://www.singlestore.com/blog/distributed-sql-workspaces-...
[2]: https://www.rockset.com/blog/introducing-compute-compute-sep...
It simply wasn't possible a few years ago (SSDs weren't fast enough and Git was too slow for projects with huge number of files).
I'm having fun with it.
Current version is just written in Javascript but if there demand hits a higher level would likely write a version in Rust or Go.
If anyone has any pointers to similar projects I'm all ears.
I have been following Dolt. Been a while since I took a look. Seems like they are making progress!
* databases need to get better yet at schema management and workload isolation to enable multiple applications to properly integrate through the database (as traditionally envisioned)
* HTAP seems inevitable but needs built-in support for row-level history/versioning to get maximum benefit
* databases should abstract over raw compute infrastructure efficiently enough that you don't need k8s to run your application logic and APIs elsewhere. The database should be a decent all-in-one place to build & ship stuff
(Or maybe there is one but I don't know about it.)
AWS made a serverless MySQL.
There are many.
> AWS made a serverless MySQL.
AWS Aurora Serverless has both MySQL and Postgres flavors.