It minimized complexity (moving parts, boundaries, etc...), everything was transactional, and a single dump of the database gave you a copy of the entire persistent state of your system at that moment (across all services).
We had several services running, but any notion of persistent state was stored in a single database. We didn't have particularly high demands, but processed ~140M jobs per month in addition to our normal query load.
Postgres handled this like a champ, had far lower p50 and p95 latency than SNS/SQS, and rarely went above single-digit percentage CPU usage on a fairly cheap DigitalOcean box.
I've worked at a lot of big tech companies (Google, Microsoft, Twitter, Salesforce) in many varieties of giant distributed systems and the most valuable lesson I've learned over and over again is: Distributed systems are hard, avoid them until you can't.
That's the best advice I can give to any startup. Until you're regularly sustaining 1,000's of TPS or have grown to dozens of TB of data, it is generally a distraction to even think about anything other than a single relational database.
A bit outdated now, but some details here: https://paul.copplest.one/blog/nimbus-tech-2019-04.html#tech...
The vast majority of companies and their scale will never go above a single midsize database server. All of the transactional, backup, and querying functionalities you list make it much more productive than using fancy AWS/cloud services.
thank you. May I borrow this?
We wrote a very simple task queue (<500 lines of python) using a postgres table and inbuilt pub/sub. Ran millions of jobs, scheduled and unscheduled, without a hitch.
It only starts to leak if everyone accesses the the DB through raw sql, mishmashing modules and creating insane dependencies.
Give each module ownership to a table and demand to only access it through there and the backend storage system becomes irrelevant
I believe you're being deeply naive about modules owning single tables. You forego relational integrity and almost the whole power of relational algebra if you take that approach. That's heaps of functionality to leave behind when you and one or two other guys need to crank out a feature a day.
I can see how you misunderstood me there however, it was admittedly poorly worded.
What I was talking about was basic data hygine as it's often called. If you need specific data, you define a clear way to get this data and only access it through that API. This can be a class, method or anything your language of choice prefers.
If you skip that step and directly access the DB everywhere in your code, you'll create another unmaintainable dumbsterfire as soon as your team goes beyond the initial programmers.
A startup needs all the options on the table because the alternative is they go out of business and there is no startup. This idea that we can enforce “good architecture” (subject to interpretation) with technology by isolating junior engineers with networking rules needs to at least be more transparent about its motivations.
For pieces to be able to be carved off, they already have to be autonomous.
What usually happens is that devs without experience in service architectures presume that service architectures are probably just the things they are used to seeing, ie: monolithic entities.
It's usually then that we hear things like "extract the product service". The problem is that product is an entity, and entities are the last thing to build services around. That's how we end up with distributed monoliths, and subsequently failed service architecture projects.
In order for an app to be able to transition to services, the app has to be designed this way from the beginning.
And yes, it's definitely possible to know what the model partitions should be up front. They're very natural divisions. But they can't be arrived at by looking through an entity-centric lens. And unfortunately, forms-over-data apps very rarely provide us with an opportunity to learn about the "other" way to do it.
Edit: Fixed a typo
I've heard of him before but never really read anything much by him. Is this guy for real? He worked at one startup, got angry about a database, became a consultant, and screams about everyone being wrong about everything?
* I don't see any problem with stored procedures. They can make good sense for, say, auditing, as well as for performance.
* People describe their software systems as "Using Oracle" because it matters, not because their design is stupid. It tells me that Oracle skills are relevant, for instance.
* NoSQL is generally, in my experience, awful and chaotic, rather than liberating. Lots of good work has gone into making serious grown-up relational databases. Schemas, normal forms, constraints, rigorous work on the ACID properties. Something like MongoDB is just sloppy amateur-hour by comparison.
* In the YouTube video, he says that solid-state drives render relational databases obsolete. This strikes me as absurd. Not even ultra-fast SSD storage technologies will do that. The relational model is effective, and the associated DBMSs still well justified. Joins belong in a query language, not in imperative code. Why would you want to try managing a huge complex dataset manually?
[0] https://en.wikipedia.org/wiki/3D_XPoint
(Edit: formatting)
Something like VoltDB but there are other solutions. There used to be some blogs about their architecture and how they could avoid a couple of steps a normal RMDB needs that takes a lot of time.
I sincerely doubt that.
For it to be useful in real world situations, you'd have to build your own highly flexible, scalable, rock-solid, high-performance data-management solution, ideally one which enables the user to use a declarative means of expressing queries.
This is, of course, a DBMS.
If you build one which imposes structure on the data, you've got either a relational DBMS, or an object DBMS, or some other kind of well-studied database solution... except you've rolled your own from scratch, without input from database experts, so it's going to be a disaster.
Unless you're someone like Microsoft, Google, or Amazon, you pretty much can't build one of those. It costs tens of millions. It takes a huge amount of testing, for obvious reasons.
I really don't see any argument for not using a mature database system for managing a typical company's data.
Of course, if we aren't talking about a full-scale database that has to cope with a huge amount of messy mission-critical data in a changing business environment, then the game changes, and sure, you might have a chance just writing something yourself.
Netflix's video-streaming CDN ('OpenConnect'), for instance, obviously isn't powered by an RDMS. They put in a huge amount of highly technical work building their own finely tuned technologies to pull data off the disk and get it to the NIC with minimal glue in between. But that's Netflix, and virtually no real companies face that kind of challenge.
Also, VoltDB looks like a streaming DB technology. Isn't that both an DBMS (rather than a means of rolling-your-own), and very niche? I don't see how it's relevant here. If it's really able to serve business needs better, then great, but it's still a complex-DBMS-as-a-product.
If you have a proprietary database, stored procedures are awful. AHEM ORACLE.
To me, he sounds more like a snake oil salesman.
He rails against a strawman where people put SQL everywhere, into views, into application logic ("mail merge" is one he mentions), and DBAs gatekeep everything.
The common sense part: no, your views shouldn't be composing SQL with string interpolation.
The moronic parts:
- He venerates application logic and regards the data model as a detail, but data models aren't details — they tend to outlive application logic.
- He holds up in-memory data structures as a platonic ideal, but doesn't address the things databases provide for you: schemas, constraints, transactional semantics and error recovery, a clear concurrency story.
If you're a startup just use SNS / Kinesis / Google pub sub ect ...
- Dealing with sec upgrade for PG and Linux
- Dealing with backups
- Dealing with HA, so settings up slaves / replica, then what do you do when something goes wrong? Do you manually SSH and do some magic?
- Network security / usernames / passwords
- ect ...
Clearly what startup don't want to do and usually lacks expertise into.
SNS, Kenesis (Kafka), Google Pub/Sub are awesome. Not the same problem/solution fit as an event store, but awesome for the scenarios and architectures they're targeted at.
I understand that Kafka can scale horizontally and can handle crazy throughput, but I mean from a programming point of view, the idea of a unified log as a data model applies to both, correct?
Event-based just means that communication happens over events. Event-sourced means that the authoritative state of the system is sourced from events. If the events are literally the state, then how those are retrieved begins to matter.
Kafka breaks down as a message store in 2 key ways that I mentioned elsewhere in all these threads.
> The first is that one generally has a separate stream for each entity in an event-sourced system. Streams are sort of like topics in Kafka, but it would be quite challenging to, say, make a topic per user in Kafka. The second is Kafka's lack of optimistic concurrency support (see https://issues.apache.org/jira/browse/KAFKA-2260). The decision to not support expected offsets makes perfect sense for what Kafka is, but it does make it unsuitable for event sourcing.
If my only tool were Kafka, then I wouldn't be able to use the messages in the same way that I can with something like Message DB. And that's okay, different tools for different jobs.
I've been building systems on some pretty similar ideas for the last 5+ years, and have a similarly optimistic outlook on it. Just to add another datapoint here, we've been able to scale them up to 20-40x higher than your figure (a couple thousand jobs/sec) on a single (albeit beefy) MySQL instance. It definitely takes a fair bit of care & we certainly have some time invested in it, but it can be done.
I guess my services haven't had (gotten to? :-) ) to share databases with other services for quite some time -- they did when I started my job! -- but even with that concession, the benefits of DB guarantees have been amazing. There's nothing like a transaction for preserving your sanity.
The best tactic I've learned from these systems is to think really hard about the batching & partitioning behavior to keep msgs/sec high enough & transactions/sec low enough to stay afloat.
Scalability for a tech product should be a concern at some point, but if you don’t have product market fit it doesn’t matter. The complexity of building a distributed system right out of the gate will cripple rapid iteration or lead to a broken product.
The first is that one generally has a separate stream for each entity in an event-sourced system. Streams are sort of like topics in Kafka, but it would be quite challenging to, say, make a topic per user in Kafka. The second is Kafka's lack of optimistic concurrency support (see https://issues.apache.org/jira/browse/KAFKA-2260). The decision to not support expected offsets makes perfect sense for what Kafka is, but it does make it unsuitable for event sourcing.
Being built on top of Postgres, Message DB gives you access to event sourcing semantics using a familiar database technology.
We use Message DB in our production systems, and I'd be happy to talk more about it if you have other questions. We've found it very reliable.
As a disclaimer, I'm listed as a contributor at the Eventide Project, the project Message DB was extracted from, though I did not write any of the code behind Message DB.
Some clarity by example: In financial systems, it’s extremely important to keep track of all the transactions between your microservices (if your architecture is based on that). You could potentially lose a message delivered to you via a broker if your service fails to write it to a persistent storage. If the producer of that message never stored that message or it was produced on wire and transmitted, there is no way to recover it anymore. A system designed around a Message Store can mitigate such problems. You can build that similar architecture with brokers as well but for each of your application, you will have to implement something analogous to a Message Store to handle idempotency and things like that.
Polling sounds very crude, but for the systems Message DB is designed for, it's a virtue. No back pressure problems, for example.
Atomic changes (rollback can undo any unprocessed messages)
Fewer moving parts, if you already need a DB
Easily query state of queues and messages with famailar SQL