Ok, but does it scale?
spacetimedb.com
spacetimedb.com
I used to work at Cockroach Labs. The problem it's solving is fundamentally different. CRDB as a solution makes sense when you need to _guarantee_ that transactions are serializable and durable, and that your application can survive node or region failures while maintaining consistency. In a naive deployment it's significantly slower than operating on a single core, but that's the price that you pay for the ability to survive node loss without data loss.
I don't see anything that indicates how spacetime solves the core problem CRDB does, which is guaranteeing that single node failures can be tolerated with zero data loss or loss of availability. It sounds like transactions by default are required to be written to disk before completion, which makes them durable on a single node, but you can't ensure they're consistent across nodes without accepting the network overhead and losing transaction throughput (on writes, anyway).
Also, FWIW, in the several years I worked covering basically every incident, I can't recall seeing a network-bound cluster. Like anything else, there are tradeoffs. You give throughput, you get consistency and availability, and you don't need to engineer how to avoid data loss or availability with node failures. Unless I'm misunderstanding, spacetime is solving a totally different problem.
They are yolo mode by default with periodic fsync and a big mutex around every reducer: https://strn.cat/posts/spacetime/ (granted things may have changed since that blog post)
> I can't recall seeing a network-bound cluster
I saw some of these (most packets per second not bandwidth) in the Firebase Realtime Database because changes get broadcast to many users. Since SpacetimeDB is made for games this is the same synchronization effect. Traditional databases don’t do this which is why Cockroach wouldn’t have seen it.
Reposting what I posted below regarding the strn.cat article:
I'm a cofounder of SpacetimeDB (and the author of OPs article). The https://strn.cat/posts/spacetime/ article has several substantial errors. I've spoken with Vicent directly about them.
Most notably, almost the entire commentary about durability is incorrect. SpacetimeDB does not acknowledge anything before data is fully persisted to disk, even though he claims it does. Clients CAN chose to listen before that, but you can do the same thing in Postgres if you want.
There is no 50 ms delay to writing to disk. The article is mostly nonsense.
Ask Claude yourself: https://github.com/clockworklabs/SpacetimeDB
He spent 15 minutes looking at our code (by his own admission), having never written a database storage engine before AFAIK, and made a pronouncement that SpacetimeDB wasn't a good database. Crazy stuff.
On the SpacetimeDB Standalone (single node) version, you lose your data, same as you would with Postgres or Sqlite.
> I don't see anything that indicates how spacetime solves the core problem CRDB does, which is guaranteeing that single node failures can be tolerated with zero data loss or loss of availability
Multi node systems tend to have more failure modes, so fail more often, high availability is not free.
If you replicate data between two nodes (that are note async backups). You have to maintain strong data consistency. So writes to the database are considered successful only if the data is written to both nodes. If either node became unavailable you'd be unable to write anything without sacrificing consistency, reducing our overall availability rather than improving it. Also increasing latency.
This is why the whole multi node HA push by the industry is so laughable. A lot of time it makes systems more fragile.
We solve this with distributed state machine replication. You don't need multiple writers to solve the single node failure problem. You only need multiple writers for a write throughput scaling problem.
They're separate problems.
I think there's alot of interesting things happening in the database space. Vitess/Neki, vector stores, spacetime are all really good things to be happening. I think it's a shame that database developers seem to have a drama filled timeline out there. It's... all very exciting, together.
Anyway, Some things that need improvement (some of which are addressed by this blog post):
1. Backups (fast recovery) and Disaster Recovery (slow, durable recovery)
This is a big one, but sometimes speed is not the only objective you have to meet. You need to have certainty that, if everything goes down that you (eventually) can bring things back online. I don't really have a way of doing that today.
2. Read replicas sure would be nice for analytic workloads
3. Durable writes to s3 would be nice for intermittent bursting workloads (like ci systems)
I think the Spacetime folks have their work cut out for them, not necessarily because of the technology (that's hard too) but because the AI models have seemingly decided that Neon and Postgres are all that exists.
I think spacetime has a bright future ahead of it, and I am wishing the team all the best as they work hard to imprint something new on the universe.
I would add a few other things to the list of scaling problems. Some SQL features are hard to scale out (auto_increment/serial columns, unique secondary keys, foreign keys, etc.). Some SQL query operators are hard to scale out for OLTP queries that want low latency and high throughput (DISTINCT, LIMIT/TOP-N, non-collocated joins). I take it SpacetimeDB is a nosql database, so these problems are less important to them?
As to how spacetimedb plans to scale out, I didn't follow it fully. It's hard to take the spacetimedb folks seriously (see: https://strn.cat/posts/spacetime/).
Most notably, almost the entire commentary about durability is incorrect. SpacetimeDB does not acknowledge anything before data is fully persisted to disk, even though he claims it does. Clients CAN chose to listen before that, but you can do the same thing in Postgres if you want.
There is no 50 ms delay to writing to disk. The article is mostly nonsense.
Ask Claude yourself: https://github.com/clockworklabs/SpacetimeDB
He spent 15 minutes looking at our code (by his own admission), having never written a database storage engine before AFAIK, and made a pronouncement that SpacetimeDB wasn't a good database. Crazy stuff.
- Returning it as a result to a SQL query - Sending it to clients as part of a subscription - Or a return value to the caller
The guy is currently a micro-celebrity for the storage engine he wrote: https://cursor.com/blog/git-at-any-scale
I have doubts that a global readwrite lock around a hashtable makes for a good general purpose storage system.
Calling it a "hashtable" is something that only someone who hasn't built a DB engine would do. It's incredibly naive. It discounts the complexity of execution, atomicity, durability, constraint validation, migrations, query planning, incremental query evaluation, down to zero. It really makes it sound like he has absolutely no idea what he's talking about.
Besides, it's a btree (heh).
> We spent a lot of money finding out that a lock is more performant.
I want to hear about that journey.
> Besides, it's a btree.
I guess it's not a hash table, but I think the point of Vincent's post is still worth exploring. You and he both say that essentially a key/value store is mutexed, and the user's code runs inside that mutex. That is fascinating. Why would that be better? Clearly it is, or you wouldn't have spent the money. What controls did you put in place to ensure user code didn't blow up the performance, or did you even feel the need for such controls? Is WASM VM execution fast enough for this? Is there even a market for that? Who pays to put their code inside that mutex and why? I want to know more.
The TL;DR is that because you're not holding locks across the network (as is the case in Postgres), your server code can complete transactions in single digit microseconds, rather than milliseconds. And the practical effect is you can do many more transactions per second as a result.
(I mention this mainly as a keyword that people can look up for more information)
I think one of the maybe less talked about benefits of distributed SQL is support for nearly transparent rolling upgrades of the database with very little impact to a running workload. Spanner is best in class at this.
> You may make use of the Licensed Work provided your application or service uses the Licensed Work with no more than one SpacetimeDB instance in production and provided that you do not use the Licensed Work for a Database Service.
Therefore, as an OSS product, SpacetimeDB does not scale.
https://github.com/clockworklabs/SpacetimeDB/blob/master/LIC...
Their license is basically "use freely unless our community decides you violated the code of conduct, for three years"
We hoped for tens of users.
Mongo DB is web scale
At my first software engineering role the system I 'inherited' (The guy who wrote it quit after it launched, I was hired on after) was only used by 5-10 users at a time. On the flipside, one or two of those users would do things that the system was advertised as being able to do, but would really stress out the system and slow certain operations down for everyone, including exposing our company to financial penalty risks. We actually for a time slowed down trying to get large customers signed up, because even the business folks had to manage enough complaints that they didn't want to risk damage to our reputation. [0]
At my second job, well, I remember the debate I had with the manager about a change I suggested to improve memory allocation on a new email templating process he wrote. He then gave me a lecture about 'premature optimization'. Then when the thing kept crashing with OOM errors he cribbed the parts of my changes that solved the memory leak but left out the pooled buffer to save face. [1]
[0] - The thing was written such that we could only process 2 messages concurrently from a given integration partner at a time, we eventually fixed it by moving the processing to actors behind a consistent hash router.
[1] - The pooled buffer changes were 5-10 extra lines total, lol.
This article hooked me with the comparison at the beginning.
I used to scoff at redis' single threaded design but it makes sense in a memory bound db. This article is a great example of taking that high-performance approach and designing parallelization around it. SO COOOL.
Also, it's SO interesting that here's yet another example of how performant the actor model can be. It's an old design (Communicating Sequential Processes was published in 1984!) but it works so well in our current hardware.
From a developer's perspective actors are very easy to reason about. I'm curious to try out Spacetime in a project now. Organizing server logic into databases, sub-databases, tables and reducers is intriguing.
Data comes in, do something with it, store your state in a principled way, repeat. Add in subscriptions to data and it becomes reactive. Simple.
This is a VM with 64 vCPUs running at 3.5GHz, with 256GB RAM.
It presently costs about ~3USD per hour to run.
Then we actually founded a startup and realized this was not a silly question. But perhaps that's circular when the default path is VC funding, hmmm.
https://github.com/philippta/postgresql-cockroachdb-benchmar...
> The most significant (but not sole) reason SpacetimeDB is faster than other backends is that we have decreased the round trip time between your server and your database by at least 99.95%. In SpacetimeDB your application server, ORM, and database are merged into a single system, so all three of these are run within the same process. [1]
For most systems that is a fair assumption to make but SQLite blurs the lines by running in-process. They admitted that this makes a significant difference on X [2] but do not list those results on their benchmarks. It would ruin their "is that the X axis or is that the competition" line to include. I'm hung up on this because they market it for use in video game servers, with an emphasis on MMORPGs (they're making one). Literally nobody is using web APIs for networking in real-time multiplayer games. The benchmarks are completely meaningless for that use case.
On the topic of scale they have been testing sharding/IDC for a while in BitCraft Online (their MMORPG) but they peaked at less than 5,000 concurrent players [3]. That's small enough to comfortably run on a single machine. Especially if you know that their gameplay is 90% (or more) just waiting on timers to finish. The timers only progress when you're connected to the server so that CCU number includes all the players waiting on timers.
[1] https://spacetimedb.com/blog/benchmarking [2] https://x.com/spacetimedb/status/2027766319462904310 [3] https://steamcharts.com/app/3454650
> The benchmarks are completely meaningless for that use case.
I wouldn't say completely meaningless, but it isn't a game benchmark that's true. We feel very comfortable that it's the most performant backend for persistent games though. We were trying to show it's also more performant for web use cases.
> 90% (or more) just waiting on timers to finish
This is not correct. It's mostly processing player movement transactions. We do about 50 million an hour.
Depends how persistent you really need it to be. If rolling back a few minutes is tolerable in exceptional scenarios then I disagree. It will be more performant to keep state in engine and snapshot whatever you need to save every few minutes. That's pretty much what most games do now.
> This is not correct. It's mostly processing player movement transactions. We do about 50 million an hour.
I said gameplay, not processing time.
But also vertical scaling seems to be able to handle so much these days that more people probably aught to ask themselves if their application is ever likely to truly need horizontal scaling - Maybe my work is just a small pond, but I suspect we will eventually end up in a place where the vast majority of applications are happy on a single box, and only twitter scale things need to care about this stuff.
.. or maybe I lack imagination of what possible future applications might make use of such massive horizontal scaling on such capable individual boxes (seriously).
I feel like this isn't a thing when it comes to applications. All the decisions you'd make to allow an app to scale sideways are just, like, good design decisions? Even if you never went 1->2 instances. Otherwise every app would just be PostgREST on top of a beefy VM.
We're already there IMO. A single box can scale up so much more than people expect these days - over a thousand cores and terabytes of RAM. The people coming up with overcomplicated architectures and mandatory high availability get in the way of it.
I’d like to recommend If you are really need to scale out soon, your domain is difficult to separated, only use distributed DB.
It is true, distributed databases superior to many people technically, But that’s not mean they are better in your service situation
I think it is better spend your time, separate domain architecture. Simply use DB. Do not put unpredictable future off DB as past Oracle DB handle all.
The direction is more games-to-general-purpose than games-to-AI.