PranaDB – A distributed streaming database, designed to be horizontally scalable
github.com
github.com
Instead of working on making the problem solvable with an exsisting tool.
Personally, I think it would be cool to write my own database as a side project. I would learn a lot. It is a cool thing to work on.
It is also -hard-. Really -hard- to do it. There are millions of odd edge cases that preexisting mature solid database systems have implemented over time as they became aware of them.
Usually, the shortcuts implemented in geen field new databases to make them fast or easy to use end up being impediments laster on down the line.
All of that said in context of being a database being developed by a business to run their own business on. it is, or it will become a giant time consumer. Or you can redesign the code of the actual product to fit the quirks of the homegrown DB.
At that point redesiging the code to fit an existing product will seem like a pretty good idea but now you have sunken cost syndrome.
With all that said, I value diversity. Sometimes something brilliant can come out of doing things that are a bit crazy.
I do wish that a lot more operting systems were created. It could be argued that when you might build a database to solve a product domain, a custom operating system would also get you part of the way.
Most people when they hear "oh you should create your own operating system for that" will tell you, you are nuts. (for good reason). i think the same reaction some come with "You should create your own datasbaase system"
If there was a DB that got popular that focused on the following usecase and covered it really well I think half of the worlds backend programmers would soon be without a job due to the increased efficiency in the sector:
- Focus on persisting insert-only events and defining aggregates / materialized views (event sourcing); with all the power of SQL for queries.
- Allow DB transactions where the current state of DB is queried (to check if the operation is allowed) and if it is, insert the event.
Our team programs in this style already with a traditional SQL, it is doable but one does notice one is working against the grain a bit ("if the DB engine only had this thing implemented..."). Problem is all popular DBs started out in the opposite end: That you have mutable "objects" or "documents" or "records". That is IMO a worse way to program for nearly everything.
Programming language trends have shifted from mutable and object oriented to functional and immutable. This DB is an example of the same shift for async work. But we need the same shift for transactional and consistent ACID usecases too.
I recommend the paper "Out of the tar pit" for similar ideas (though without much focus on events as such).
(PS: Yes I know about GDPR and need to delete data. The above is not an exhaustive list of features, but a description about the primary workflow of the intended audience of the DB.)
- Focus on persisting insert-only events
What do you mean? (Are there other events you don't want persisting? why? and what does 'insert-only' mean?)
> and defining aggregates / materialized views (event sourcing); with all the power of SQL for queries.
Can you explain what this means in practice?
> - Allow DB transactions where the current state of DB is queried (to check if the operation is allowed) and if it is, insert the event.
This is really unclear - what is 'current state of DB' you're interested in, what is 'allowed' here (and why)
I'd really appreciate some clarification! TIA
A database can of course never know that it is "events" you are inserting.
--
Aggregates/materialized views:
(Contrived example warning)
Say that you have a User entity in your system. The Name of the user is only set through two events, UserChangedName or CustomerServiceChangedUserData.
The "aggregate" is simply the calculation of the user's name (and other data) from such events. Declaring this should be done using some variant of functional programming; SQL views, reactive programming, etc
At least in the DB we use, we can declare a SQL view that fits the bill and does such a transform. So User is a view on the events that affects it. But it can not be materialized if it is too complex (materialized = updated when the insert of events happens, so that one does not need to query all the events every time one reads User).
One can do it using triggers, but that is an imperative instead of declarative programming style with all the problems and lack of guarantees.
--
Operation allowed etc:
Say that there is a limit of 5 name changes. However a TooManyNameChangesReviewedByCustomerSupport can be inserted and then you get 5 more.
So user wants to change name. You need to read NameChange events and TooManyNameChangesReviewedByCustomerSupport events, compute whether a change is allowed or not, and only save the event if it is allowed to happen (at which point it becomes the event -- this DB is the source of the event).
What you need now is a transaction that allows you to avoid race conditions. Otherwise a parallel thread could be doing the same thing at the same tim, and you end up jumping from 4 to 6 name changes, when 5 should be the limit.
SQL DBs focus on providing transactional guarantees for UPDATEs, but not so much on guarantees for "only INSERT this row if these other rows in another table are NOT present/has a count less than --" ..
So you end up having to update an aggregate manually in the SQL transaction to avoid races => this is what should be built into the SQL server. (It would just be syntax candy in a sense, saving one from writing code to update the aggregate explicitly, but pretty important syntax candy as it changes the mindset of the developer in an important way)
I'm nowhere near doing what you want but I've been looking for generally useful DB constraints that currently have to be done via triggers in SQL for future work (your syntax candy, basically). Have you an email contact? (I don't do linkedin, I'm very luddite). If not interested just ignore this. Thanks
(Also been wondering about how automatically-maintained-summary aggregations might be needed for businesss, and how to do them - gaian, this is very much future stuff though)
I am itching to start piping around change data capture Debezium logs so tell me now before I inflict a horror on my co-workers.
that's a new one
I’m looking for something that is stateless itself and stores data in S3 or something. We have a few topics that would be nice to explore and get stats on without having to implement our own query language.
Seems like they could’ve used that instead (plus a Kafka connector) and benefited from a lot of deep research into how to do this optimally
Nonetheless cool product, thanks for sharing and making it open source.
Patching an existing DBMS means your changes have to mesh well with all its other features. And since you have a lot of talent on payroll already, it's probably easier to write a DBMS that solves your narrow use case, but solves it 100%.