Other than batch jobs, I can't think of a problem that can be solved these days that doesn't also require high availability - at the very least they require a warm standby.
(modern server hardware and operating systems are also surprisingly reliable nowadays, which makes it harder to reach breakeven with a distributed design)
P.D: the business world is the one that needs AND tries everything, and most attempt to implement something is much worse than the simpler previous thing.
For most that I know and most I bet (even without knowing) add "high availability" or "warm standby" WILL CERTAINLY lead to worse availability.
In fact, the best setup for most people, and consider things from today (that you will see, not that different from mainframe days!):
- Single server on a *nix LTS/immutable distro with just the RDBMS + App backend, on decent hardware. Fast, lots of ram or whatever optional
- A true reliable backup system
- A decent network setup, ideally a VPN one for connect everybody, with good firewall, SSL and that is
More complex than this and is 90% certain is much worse, other 8% until the dude that do it and care leaves.
In the flip side:
> the problem with a single system is that mundane things cause unavailability
Is multiplied by each "system" you add. The basic failures are relative easy to deal and understand, the ones introduced by more complex system who knows?
Probably the most important step that I miss:
> How long would it take to procure a new server and restore from backup?
.. and applied to any complex extra you have.
For single system is viable (with something like nix) to go off for maybe half hour for what I see around, most of the time in procuring another machine (that what people do is to Bring any other machine it can not go to amazon and buy!) and restoring the backup.
Of course, I factoring that downtime is not "seconds or minutes" here, but neither I think many can do like that
P.D: all
> Is multiplied by each "system" you add
Was still the take away. The idea is the lost a single instance should cause downtown so it’s not multiplied. IE three instances of the database case tolerate a loss of a single instance for maintenance since the other two will take over the load.
There is of course an argument against the cost of this and sure I’d even accept the complexity argument since you probably need to add more tooling to manage the hand over but again the point of this complexity is specifically to avoid having a single point of failure / to naturally handle failure such that the possibility of downtime isn’t multiplied
McSherry does a lot of interesting work on making monotonic/incremental distributed systems efficient (e.g. Differential and Timely Dataflow). Those kinds of systems scale much more linearly.