Errr yes. Without much sweat really.
Just because something started ~30 years ago doesn't mean it hasn't updated with the times, and doesn't mean it was built on bad foundations.
Without much sweat for general purpose workloads.
But transaction processing tends to have power law contention that kills SQL row locks (cf. Amdahl’s Law).
We put a contention calculator on our homepage to show the theoretical best case limits and they’re lower than one might think: https://tigerbeetle.com/#general-purpose-databases-have-an-o...
In fact large real world systems are not limited to 100-1000 TPS, or even 10 kTPS as the calculator tries to suggest. That's not because Amdahl's law is wrong, the numbers you're plugging in are just wildly off, so the conclusions are equally nonsensical.
There might be some specific workloads where you saw those numbers, and your DB might be a good fit for this particular niche, but you shouldn't misrepresent general purpose workloads to try to prop up your DB. Claiming that SQL databases are limited to "100-1000 TPS" is unserious, it is not conductive to your cause.
> Without much sweat for general purpose workloads.
But writing that traditional SQL databases cannot go above these "100-1000 TPS" numbers due to Amdahl's law is going to raise some eyebrows.
I don't think that's controversial. Amdahl's law applies to all software. Its not a peculiar feature of SQL databases. The comment is well-contextualized, in my view, but reasonable minds may disagree.
TFA contextualizes this better:
> This gets even worse when you consider the problem of hot rows, where many transactions often need to touch the same set of “house accounts”.
I think this is very clear, I don't know why you're saying that tigerbeetle is trying to make a generic claim about general workloads
The comment you're replying to explicitly states that this isn't true for general workloads
It still holds up basically whole of internet.
in most cases SQL is good enough for 90% of workloads.
Plus micropayments, of course :-)
But we couldn't get rid of it because it papered over something important.
config defined in YAML.
when I woke up today, I didn't really expect to be convinced that we live in a relatively good timeline, but...
The combustion engine's fundamental design is pretty damn good and that it had to be updated to handle unleaded gasoline isn't a knock (pun... intended?) against that design.
Sometimes I feel like we software engineers have the worst memory of any engineers.
Actually, load balancers are a great example. The number of times I’ve seen a team re-implementing HAProxy, but poorly, is entirely too high. The reasoning is always something along the lines of “we need this extremely specific feature.” OK, and have you even read the manual for HAProxy? Or if that feature truly doesn’t exist, did you consider implementing that separately?
Some months ago I was re-enlightened when Anders Hejlsberg (creator of C# and TypeScript) explained why they chose Go for reimplementing the TypeScript compiler, instead of using any of those languages or something like Rust.
The way they defined the problem, the tests they did and how they justify their pick is how these kind of decisions should be made if we want to call ourselves engineers.
The relational model has shown itself to be exactly the flexible and powerful model that Codd said it was, even in its relatively-debased from in SQL.
In fact the potential of it as a universal and flexible data model that abstracts away storage still remains to be fully unlocked.
Now as for existing SQL databases, yes, many of them were built on foundational assumptions about the nature of memory and secondary storage that no longer hold true.
Many of us in this industry still have our heads stuck in the past world of spinny spinny magnetic disks on platters with heads and cylinders and grinding noises. Real world hardware has moved on. We have a couple orders of magnitude higher IOPS generally available than we did just 10-15 years ago.
So I'm excited about products like CedarDB and others (Umbra before it, etc) that are being built on a foundation from day one of managing with hybrid in-memory/disk.
Throwing out SQL is not really the recipe for "performance" and the lessons of separating the storage system from the data model is still key, since the 1960s.
I am willing to grant that a specialized transaction/ledger system like TigerBeetle might have its place as an optimization strategy in a specific industry/domain, but we should not make the mistake of characterizing this as a general problem with the relational model and data storage broadly.
We've discovered hacks to work around the limitations of SQL, so you can maintain performance with sufficient hackiness, but you have to give up "purity" to get there. Worse is better applies, I suppose, but if we were starting over today the design would be very different.
Throwing out SQL isn't strictly required, but like Rust isn't strictly required when you already have C, sometimes it is nice to take a step back and look at how you can improve upon things. Unfortunately, NoSQL turning into a story about "document databases" instead of "A better SQL" killed any momentum on that front.
SQL was the first serious attempt to translate this into a working system, and it's full of warts, but the underlying model it is reaching towards retains its elegance.
But most importantly the principle of not tying the structure of data to its storage model is absolutely key. Not least because we can iteratively and flexibly improve the storage model over time, and not be stuck with decisions that tie us to one particular structure which is the problem that 1960s "network" and "hierarchical" databases (and modern day "key value" or "NoSQL" databases) cause.
Within a single machine, yeah, relational dbs still work like a charm.
If you were to design an OLGP DBMS like Postgres today, it would look radically different. Same is true for OLTP.
Databases have not been bottlenecked on storage bandwidth in a long time but most databases are designed as if this was still the case. Optimizing for memory bandwidth, the current bottleneck, leads to substantially different architectures than are commonly deployed.
But our customers need separation of concerns in their architecture. While they could put the cash in the general purpose filing cabinet, they actually want the separation of concerns between OLGP system of reference (string database, i.e. PG) in the control plane, and OLTP system of record (integer/counting database, i.e. TB) in the data plane.
But for anyone tempted by Oracle, do remember that the upfront, agreed licence costs are only a fraction of the true price:
You’ll need someone who actually knows Oracle - either already in place or willing to invest a serious amount of time learning it. Without that, you’re almost certainly better off choosing a simpler database that people are comfortable with.
There’s always a non-zero risk of an Oracle rep trying to “upsell” you. And by upsell, I mean threatening legal action unless you cough up for additional bits a new salesperson has suddenly decided you’re using. (A company I worked with sold Oracle licences and had a long, happy relationship with them - until one day they went after us over some development databases. Someone higher up at Oracle smoothed it over, but the whole experience was unnerving enough.)
Incidental and accidental complexity: I’ve worked with Windows from 3.1 through to Server 2008, with Linux from early Red Hat in 2001 through to the latest production distros, plus a fair share of weird and wonderful applications, everything from 1980s/90s radar acquisition running on 2010 operating systems through a wide range of in house, commercial and open source software and up to modern microservices — and none of it comes close to Oracle’s level of pain.
Edit: Installating Delphi 6 with 14 packages came close, I used 3 days when I had to find every package scattered on disks in shelves and drawers and across ancient web paces + posted as abandonware on source forge but I guess I could learn to do that in a day if I had to do it twice a month. Oracle consistently took me 3 days - if I did everything correct on first try and didn't have to start from scratch.
I especially remember one particular feature that was really useful and really easy to enable in Enterprise Manager, but that would cost you at least $10000 at next license review (probably more if you had licensed it for more cores etc).
What I wrote about above wasn't us changing something or using a new feature but some sales guy at their side re-interpreting what our existing agreement meant. (I was not in the discussions, I just happened to work with the guys who dealt with it and it is a long time ago so I cannot be more specific.)
You can run a hell of a lot off of a small fleet of beefy servers fronted with a load balancer, all pointing to one DB cluster.