Announcing SurrealDB 1.0
surrealdb.com
surrealdb.com
and written by someone who should have spent more time studying the history of database systems.
> Imagine a world where the majority of your backend logic is seamlessly embedded within the database itself
This is not a good idea. It has been done many times and never quite caught on because it is not a good idea.
From a security perspective it is a nightmare.
Or if you do put all the correct isolation around the code to protect the database, then you have basically created an "app server" (old term) inside a database, and it would happily, run outside of the database since in essence it is already doing so.
It was quite normal in 70s 80s and even 90s; all the ms sql, db2, oracle and as400 systems I encountered in those days had all or almost all logic as stored procs. Very large ones.
Largely thanks to NeXT showing the way with EOF (the original ORM).
Replacing 3 classes, and numerous methods with two 6 line queries that does the same task in a thousandth of the time is quite satisfying.
Stored procs might be considered evil; they did make the developer acutely aware of the innards of the database instead of just not even knowing how a db works and which things are expensive. To know that you need to look past the leaky abstraction of ORMs and that's distracting from making features.
I have heard tech leads in this situation claim that because everything runs fine in the cloud without knowing anything about postgres/mysql/dynamo etc, and those extra costs are cheaper than people, it's fine. But our team wasn't hired because it was fine; we were hired because hosting costs were eclipsing the developer costs. It's not that hard to do when stuff is just incredibly badly built...
Another big issue I have with ORMs is that a lot of them are hostile to you writing queries in the first place. Network IO is the worst, please stop trying to force us to use your terrible overlay.
It is painful looking at modern software development. They don’t know the basics and the whole Agile/TDD thing I think was the start of the end because table schema (and every other kind of planning) is not really compatible with agile.
Instead, we haul that shit to the NIC, then across the network, then copy it into memory on some server (probably inefficiently), do the operations there, or we have to reinvent this with pushdown functions for distributed databases.
There are many cases where moving function-to-data is the right answer.
You can drastically cut down on end-to-end latency by eliminating the network hop from your app server to your DB, especially in cases where you are forced to make multiple back-to-back DB queries per request.
I think these conversations usually boil down to some kind of pragmatism versus idealism, so let me be frank about that. If everything could be done as close to the data as possible, that would be faster than not. Every boundary / transformation is a few burned cycles and delays waiting for the arrival of photons/electrons. The ideal here, really, is to have everything occur in one process on one machine without hitting the disk. Ideally the User's machine, for whatever value of User.
However, there are other considerations which cause us frequently to draw the boundaries differently. Often just as valid.
Where certain things happen is relatively flexible, it's software and we can come up with whatever architecture we can come up with. If you're moving more logic into the database process, then there are costs to that but they're not measured in cycles. Others if you do it by moving the database into the application process.
Maybe those costs make sense, in which case go for it. We've had the means to do arbitrary work in the database for many decades, and for some periods it was even fashionable.
> Advanced inter-document relations and analysis. No JOINs. No pain. […] queries allow for multi-table, multi-depth document retrieval, efficiently in the database, without the use of complicated JOINs
That sounds like they have a networked database, not a relational database. That might be fine, but as I understand it, relational databases won because they offer more flexible access patterns and it’s easier to write correct queries.
https://github.com/surrealdb/surrealdb/blob/ed60a35b9b539e1b...
I could be missing something from their branding but I'd rather see projects with non-open source licenses from the beginning that bait-and-switch license change when they want to become profitable. (Although, I'd much rather just see a fully open source project)
They have a page on their website where they claim to be "an open source company" (https://surrealdb.com/opensource)
thinks of Oracle APEX
thinks of IBM AS400
shudders
API has a cool design though.
The dev team needs to look at perf before the complexity hole gets too insane if it hasn't already.
I know the license is okay, but from my reading of it, sounds like if SurreaDB really wanted to shut you down they could.
Otherwise, I find the architecture of it to be beautiful and compelling.
A “Database Service” is a commercial offering that
allows third parties (other than your employees and
contractors) to access the functionality of the
Licensed Work by creating tables whose schemas are
controlled by such third parties.
One example of this is that the database may create a directory structure per schema which will then result in file system performance degradation or hitting end user limits.
Don’t need to. Majority of my work is de-coupling systems “designed” around this idea.
Why can’t the database be the database, the backend be the backend, etc ? You can’t have a toolbox full of multi use tools. Sometimes you just need a fucking Phillips head screwdriver.
The strategy is to break the subscription up into listens based on the read-set ranges of the query. Then you put the individual read-set ranges into a system table that you index. Finally when writes happen, you notify all queries who's read-set intersects with the write-set.
For example, say I have a query `SELECT * from block WHERE parent_id = X AND last_modified_at > Y`.
This query might create two subscriptions for its read sets:
{ query_id: Q, table: block, column: parent_id, min: X, max: X }
{ query_id: Q, table: block, column: last_modified_at, min: Y, max: Infinity }
We can find our subscriptions to notify by doing queries like these:
SELECT query_id FROM subscription WHERE
table = block
AND (
(column = id
AND min <= new_block.id
AND max >= new_block.id
)
OR
(column = parent_id
AND min <= new_block.parent_id
AND max >= new_block.parent_id)
OR
(column = last_modified_at
AND min <= new_block.last_modified_at
AND max >= new_block.last_modified_at)
)
I'm sure there's a lot of details and optimizations you can do on top of this; finding the right read sets seems pretty tricky for complex queries. Plus stuff like batching/debouncing, etc.
There’s several challenges with this approach that come up for me (which unless I’m mistaken is the naive approach of checking each write against a notification set).
The first is that maintaining the read set seems very expensive since it scales with the number of live queries installed. In a multi tenant DB, that seems tricky to manage in terms of memory used.
The second is that the computation of intersection with a read range seems potentially very expensive. Imagine you have a string column. You now have to do a string comparison for every insertion that might hypothetically match the query.
Finally, computing that read set correctly seems challenging (as you mention) and it’s not immediately clear to me it’s always tractable (eg could you do this with arbitrary complicated joins?).
Additionally, in your description, each write has an implicit table scan to do the select to find the intersection. That will tank write throughput even for small total intersection sets (eg there’s a reason LSM databases do deletes by inserting a tombstone instead of checking whether the data exists first, same with the merge operator in RocksDB - a db read in the write path significantly kills performance)
To the specifics of my solution, you’re going to visit many less than O(subscriptions) rows using a btree index on min & max as I suggested, and I’m sure there’s use case specific data structures that could do an even better job.
You also don’t need to stall the write transaction until subscriptions are notified; you can batch up that work into big chunks and process it on a background queue after the transaction commits.
Finally, you don’t need to have subscribe to read sets that exactly match the query. You can simplify them a lot for those kinds of complex cases because you can tolerate over-subscription, re-running the query even if it hasn’t changed, checking the new result, and then deciding to deliver it to the external subscriber or not.
I would suspect this is what Google does with Cloud Firestore.
Batching only helps if you’re able to amortize the cost of notifications by doing that, but it’s not immediately clear to me that that there’s an opportunity to amortize since by definition all the book keeping to keep track of the writes would have to happen (roughly) in the synchronous write path (+ require a fair amount of RAM when scaling). The sibling poster made mention of taking read / write sets and doing intersections, but I don’t think that answers the question for several reasons I listed (ie it seems to me like you’d be taking a substantial performance hit on the order of O(num listeners) for all writes even if the write doesn’t match any listener). You could maybe shrink that to log(N) if you sort the set of listeners first, but that’s still an insertion speed of m log(n) for m items and n listeners or O(mn) if you can’t shrink it). That seems pretty expensive to me because it impacts all tenants of the DB, not just those using this feature…
Io also has persistence baked into it: https://iolanguage.org/reference/
But I'm sure for most of the HN crowd one has Java in the URL and the other is a esoteric cool language from the early 00's so.. not practical. :)
I worked with Madeleine for a few years and while the performance was amazing, RDBM systems are ahead because of the many years of standardisation, tooling and capabilities (e.g. migrations)
The other very important thing is - DB latency is an issue only in the hands of a novice. In the real world, network latency is a couple of magnitudes higher.
The issue is still open as we haven’t resolved all of it yet so makes a good reference.