Hacker News Reader

Comment by projectileboy | Hacker News Reader

Great list. Only thing I’d add for the other enterprise developers out there is to first default to not building a distributed system at all, but rather build a much smaller monolith. In 25 years I’ve worked for so many orgs that wanted to build The World’s Most Scalable System for what would maybe be a few hundred concurrent users. Not surprisingly, those projects tend to tank.

View on HN

Hear hear.

This is really important wisdom being shared.

And with a single monolith you can comfortably handle several thousand concurrent users, not just several hundred.

From my humble experience I've found that it is also relatively easier to migrate an established monolith to a semi-distributed system, than it is to build and scale a distributed system from the ground up whilst at the same time trying to figure out all its kinks before it too becomes an established system.

The trick is to find "fault lines" throughout your monolith - places where it's easy to create separation of concerns and narrow interfaces. Maintain and document these like they are external APIs. Only pass simple data through it (serialized or anything easy to serialize).

When the time comes, cleave these chunks off and wrap them in RPC/REST/graphql.

It's good form even for monoliths since it makes for an easy interface to test.

+1 to you sir. I was just having a discussion this past weekend with some technical friends and noted that a well-architected monolith often lends itself to easy deconstruction such that you get the best of both worlds when they're necessary in your product lifecycle.

Good tip, and I learnt some new terminology too ("fault lines").

I guess the trick when building new monoliths is to identify these fault lines up front - as opposed to hopefully finding them later.

Will certainly guide my development going forward, thanks.

Ah, i kinda agree. My former employer developed and sold an editorial suite architected in about 1999-2000 around the SOA model (along with corba) and later evolved through the years.

So basically some monoliths playing together.

I've always been admiring that design and how much it could do and handle with so little resources.

Most times for a whole editorial newsroom a single 48c/128gb machine can be more than enough.

They were particularly smart on avoiding relational databases and sticking to an object database (Versant oodbms, another mostly unknown marvel of software engineering).

That software, even with its shortcomings, was really a great piece of software engineering.

People today don't realize how cheap you can get a really upscaled machine. In Azure, a 48 core, 192GB machine is 2000 USD a month. This is nothing compared to upscaling your app on 10 servers and 20 services, with a lot of overhead, maintenance and dev time.

If you can get away with a monolith, it's the way to go, as long as you leave room to quickly grow horizontally as well, if needed.

> And with a single monolith you can comfortably handle several thousand concurrent users, not just several hundred.

That depends on what kind of service you offer, of course.

Haha yes, good point - nothing with video ^_^

That is a great advice. However if scalability must be introduced later on, it can be really hard as there are many features that have been added to the monolith, and refactoring it to become scalable can be a huge task.

The conditions where the monolith must be converted to a scalable system should be defined as early as possible.

Designing something to be “scalable” before it’s actually established is a recipe for premature optimization at best and a poorly baked SOA with service boundaries that make change incredibly difficult at worst.

It’s important to keep mind that SOA is about scaling teams first, code second and not really about throughout per se. A share-nothing web tier plus a couple judiciously applied databases and background job queues can effectively scale a huge proportion of applications without the overhead of a full SOA.

This too is excellent advice.

Start off with a monolith if it makes most sense (in terms of simplicity / MVP) but from the start keep future scalability in mind and plan for it.

I like this.

Unless, of course, you intend to cap users at a specific amount for another reason.

But our industry has all kinds of perverse incentives to do resumé driven development. How does one get to put paxos on their resumé if they built the correct simple solution?

The problem with scaling is that visitor count often goes exponentially. So while in the beginning everything seems calm and quiet, suddenly you can see a massive spike in interest. And if you didn't take the right measures, you might lose all that interest.

That's what startups are seeking, but it's not that often at all. Also, exponential doesn't necessarily mean "happens too fast to react". If the number of your users doubles every year, you still have exponential growth, if you only have paid users you still have a very nice growth of business and you also have time to react.

It's indeed pretty rare to see services that crumble under the exponential increase in traffic. Especially if it's not a spike from publicity (I don't know how you would say "being slashdotted" these days). It happens, but it's rare. But because it's usually newsworthy or at least interesting it gets more attention and it will feel much more of a danger than it actually is. Both for software developers and entrepreneurs (or even for the general public) it feels lame. Ha! They should have expected this! But the truth is that most of the time you should not prepare for this, because it's pretty rare, it's not even necessary for success, not even for startup scale success at the beginning.

It's pretty easy to see actually: (almost) all of today's successful services provided by startups started as monoliths (or maybe more realistically some kind of SOA, because monolith vs. microservices is really a false dichotomy). It did work a decade ago with weaker hardware, why wouldn't it work now?

There's times when you might be able to expect a huge spike, but even then, if the product is early, so the overall design is still pretty fluid, I would try to keep flexible.

It's more important to be aware of overload, and provide feedback to users (or potential users) and some form of load shedding. And test that all. For a lot of applications, being able to quickly plop together a second monolith could work to address spikes. Or switching to a bigger machine: Epyc 2-socket systems get you up to 128 cores (256 SMT threads), and I think 8TB of ram. You can do an awful lot with that.

Yep, it is still a matter of abandoning the monolith and building a new scalable system in whatever fashion you choose at that time, micro services or whatever.

In today's cloud deployments where cost is not a mega concern (as compared to having physical servers in a DC), this approach is very much feasible.

You could run two systems in parallel (monolith v/s scalable) and ditch the monolith once you have the necessary cutover logistics in place.

Of course, the sooner you do this, the better and it is non trivial to figure out the 'when'

That's great advice, as long as different systems in this monolith are loosely coupled both in terms of code and data, and you keep in mind that you will need to move them to separate services later on.

And most of the time a distributed system just ends up being a distributed monolith.

Reply on news.ycombinator.com