The best part is when it’s all architected to depend on something that becomes essentially a single point of failure, like Kafka.
There is really nothing wrong with a large vertically scaled up SQL server. You need to be either really really large large scale - or really really UNSKILLED in sql as to keep your relational model and working set in SQL so bad that you reach its limits
Sadly that's the case at my current job. Zero thought put into table design, zero effort into even formatting our stored procedures in a remotely readable way, zero attempts to cache data on the application side even when it's glaringly obvious. We actually brought in a consultant to diagnose our SQL Server performance issues (I'm sure we paid a small fortune for that) and the DB team and all of the other higher ups capable of actually enforcing change rejected every last one of his suggestions.
Large or small, it's always going to be a single point of failure. All hardware is fallible, especially if we're talking about a commodity box rather than a mainframe or something.
It works, with a lower cognitive burden than that of horizontally scaling.
For the loading concern (i.e. is this enough to handle the load):
For most businesses, being able to serve 20k concurrent requests is way more than they need anyway: an internal app used by 500k users typically has fewer than 20k concurrent requests in flight at peak.
A cheap VPS running PostgreSQL can easily handle that.[1]
For the "if something breaks" concern:
Each "fault-tolerance" criteria added adds some cost. At some point the cost of being resistant to errors exceeds the cost of downtime. The mechanisms to reduce downtime when the single large SQL server shits the bed (failovers, RO followers, whatever) can reduce that downtime to mere minutes.
What is the benefit to removing 3 minutes of downtime? $100? $1k? $100k? $1m? The business will have to decide what those 3 minutes are worth, and if that worth exceeds the cost of using something other than a single large SQL server.
Until and unless you reach the load and downtime-cost of Google, Amazon, Twitter, FB, Netflix, etc, you're simply prematurely optimising for a scenario that, even in the businesses best-case projections, might never exist.
The best thing to do, TBH, is ask the business for their best-case projections and build to handle 90% of that.
[1] An expensive VPS running PostgreSQL can handle a lot more than you think.
Not to forget: those costs are not just in money and time, but also in complexity. And added complexity comes with its own downtime risks. It's not that uncommon for systems to go down due to problems with mechanisms or components that would not exist in a simpler, "not fault tolerant" system.
That's still a business decision.
Customers don't vote with their feet based on what tech stack the business chose, they vote based on a range of other factors, few, if none, of which are related to 3m of downtime.
There are little to no services I know off that would lose customers over 3m of downtime per week.
IOW, 3m of downtime is mostly an imaginary problem.
Services that people might leave, because of downtime are for example a git hoster, or a password manager. When people cannot push their commits and this happens multiple times, they may leave for another git hoster. I have seen this very example when gitlab was less stable and often unreachable for a few minutes. When people need some credentials, but cannot reach their online password manager, they cannot work. They cannot trust that service to be available in critical moments. Not being able to access your credentials leaves a very bad impression. Some will look for more reliable ways of storing their credentials.
> The business can try to decide what those 3min are worth, but ultimately the customers vote by either staying or leaving that service.
What do you think the business is doing when it evaluates what 3 minutes are worth?
I don't understand, what people are arguing about here. Are we really arguing about customers making their own choice? Since that is all I stated. The business can jump up and down all it wants, if the customers decide to leave. Is that not very clear?
I think the point is that, for a few minutes of downtime, businesses lose so little customers that it's not worth avoiding that downtime.
Just now, we had a 5m period where disney+ stopped responding. We aren't going to cut off our toddler from peppa big and bluey for 5m of downtime per day, nevermind per week.
You appeared to be under the impression that 3m downtime/week is enough to make people leave. This is simply not true, especially for internet services where the users are conditioned to simply wait.
You saying “the customer can decide to leave” is not a counter to this at all. It’s just a weird way of saying what everyone else is but framing it as a counter argument to what is being said. Which it simply isn’t.
I've never, in my life, seen an error in SQL Server related to SQL Server. It's always been me, the app code developer.
Now, to be fair, the server itself or the hardware CAN fail. But having active/passive database configurations is simple, tried and tested.
I wish we would use something like Debian and take advantage of tech like systemd. But alas, we're still using COM and Windows Services and we still need to remote desktop in and click around on random GUIs to get stuff to work.
Luckily, SQL Server itself is very stable and reliable. But even SQL Server runs on Linux.
This is a very simple distinction and I am not sure why is it not understood
For some reason people design public apps the same as internal apps
The largest companies employ circa 1 million people - that’s Walmart, Amazon, etc. most giants, like Shell, etc. companies have ~ 100k tops. That can be handled by 1 beefy server.
Successful consumer facing apps are hundred millions to billion. It’s 3 orders of magnitude difference
I have seen a company with 5k employees invest into mega-scalable microservice event driven architecture and I was was thinking - I hope they realise what they are doing and it’s just CV-driven development
Agreed, but the cost/benefit should be the same at every level in the stack. I get it when people build a single-server system. I'm just baffled that so many people insist they need a load balancer and multiple instances of their application so that there's no single point of failure (which is not free by any means), but then run them all off of a single SQL server.
It's less true today when redundancy is baked into SaS products (like AWS Aurora, where even if you have a single database instance, it's easy to spin up a replacement one if the hardware on the one running fails).
As someone using Kafka, I'd like to know what the (good) alternatives are if you have suggestions.
Where I'm at, most of Kafka usage adds nothing of note and could be replaced with a rest service. It sounds good that Kafka makes everything execute in order, but honestly just making requests block does the same thing.
At least then I could autoscale, which Kafka prevents.
I found it easy to get up and running, even as a RAFT cluster, but I have not tried to use JetStream mode heavily yet.
One of the most stable system archictectures I've built was on Kafka AND it was running with microservices managed by teams across multiple geographies and time zones. It was one of the most reliable systems in the bank. There are situations where it isn't appropriate, which can be said for most tech e.g( K8S vs ECS vs Nomad vs bare metal)
Every system has failure characteristics. Kafka's is defined as Consistent and Available and your system architecture needs to take that into consideration. Also the transactionality of tasks across multiple services and process boundaries is important.
Let's not pretend that kubernetes (or the tech of the day) is at fault while completely ignoring the complex architectural considerations that are being juggled
From my experience most microservices aren’t engineered to handle back pressure. If there is a sudden upsurge in traffic or data the Kafka cluster is expected to absorb all of the throughput. If the cluster starts having IO issues then literally everything in your “distributed” application is now slowly failing until the consumers/brokers can catch up.