Comment by baz00 | Hacker News Reader

Comment by baz00 | Hacker News Reader

Parent Full thread

How this usually turns out...

1. We got into this microservices fad and built this really granular service architecture which we're so proud of but is actually a pile of steaming shit and we're in denial.

2. We forgot to consider that a service boundary is a transaction and/or failure domain and now we have this distributed operation that has to be transactional because it's crapping itself 50 times a day and clients are complaining that something is inconsistent somewhere.

3. Read Martin Fowler's blog and this blog article while having a post-work shit in the bathroom.

4. Come out with a half baked understanding of distributed transactions which is good enough for your ego to think is a really good business outcome.

5. Design another fucking microservice and correlation system for transaction registration.

6. Hand it over to some junior and senior developers who half-ass the whole thing because they don't understand your inane ramblings and have no idea how to test it properly.

7. Throw it into production and shout SUCCESS as loud as possible over the sounds of creaking, grinding and explosions.

How this should have turned out...

1. Actually realise that you're only doing about 5 million transactions a day and that's hardly anything.

2. Stuff the whole shebang behind a monolith with a single call, single transaction and in one storage and failure domain.

3. Pay for a slightly fatter DB instance.

4. Go back to developing things with a better ROI.

Apologies for the bitterness but it is realistic.

I know a few people who work at a place where they stuffed all their databases on the same server, and all their systems just write to whatever database they feel like on that same server, share data via database, join data across databases etc. And they use upvoted cynical memes like this comment to justify their “architecture” because “microservices are a fad”.

So as one might expect - various automated reports and actions routinely fail with deadlocks and timeouts, any change anywhere has a pretty big chance of screwing some other distant part of another system, all services need full unrestricted access to all databases, and off course nothing can really be tested because you need to run practically every database and service on the same machine to even try to test.

Naturally the pace of work is glacial, and bugs are blamed on individual intellectual failing of lesser tenured developers not having the experience and the vast system trivia knowledge of “lifers” who secured their guaranteed employment this way.

You might even be those people for all i know.

Both of these things can be true. I worked on a monolith with one database and boring, correct, transactions, and it was a joy for almost all the time. Productivity was high.

We went through a few growing pains when we had some big reports, but the first pass just moved them to a ~100 line batch reporting system, that solved it for another 2 years. Then we hit growing pains again but by that time had a shiny new data warehouse and just moved them to use the data warehouse instead.

There's truth to both sides of this, but optimising for where most engineering is happening, keeping developers productive, and simple solutions that everyone understands, have always paid off well in my experience.

Its almost as if people work on different kind of systems, and each problem and solution requires context and nuance, and when you come across an academic article describing a complex solution to a complex problem in complex system that has complex operating requirements that you have never encountered professionally and don’t have any experience with - it would be best to file it away for later, or just move on.

But maybe posting a snarky meme about it isn’t constructive.

Having multiple systems managed by different teams sharing a database is not a monolith. I've seen that idea brought up multiple times by microservices enthusiasts as a way to route around the difficulty of transactions. Most monolith enthusiasts like myself would build one service on top of one database.

Exactly, a monolith is suitable for small orgs or companies which deliver one thing. Microservices are not a fad and they solve a real world problem where you need to deliver for ex: something like a cloud platform. 10s of orgs cannot work on a monolith without stepping over each other, just the communicatin over. Also microservice enable you to scale (scale-out) the parts of the system that need scaling.

At scale microservices do help, but you get into unnecessary problems when you don't have scale and use microservices.

Microservices are just monoliths…

Not one of those. They got the microservice database isolation correct. But nothing else. Well the coffee is good and the toilets are clean.

As for lifers, everyone I work with knows how many days until retirement. Sad sad people. I am merely hopping between frying pans until I can slide out into academia and get old and rot somewhere hopefully with tenure.

Having everything on the same computer will save many network rountrips

The advantage is you know quickly rather than slowly that it’s broken.

It’s great! Abundant and regular opportunities to step in and heroically fix various fires that routinely start to smolder in surprising and unexpected places.

This is how you end up bald very quickly.

step 8: move to the next startup for a +$15k salary bump before your code ever sees heavy prod usage. Mark your resume as "expert in web scale".

So so so so true. I am haunted by the ghosts of so many of those bastards. The place is like "oh no they left - quick hire another charlatan".

Haha. We called those "promotion driven architecture". I mean if you just shove the whole thing into a monolith, no matter how appropriate, how is this going to serve as a technical artifact supporting promotion?

This dumb industry can seemingly support a large amount of charlatans and honestly I don't blame them because they can get decent money while not being bitter/miserable/ashamed and letting the weight of bad code or architecture get to them.

Step 7.5: Show all that to management who demands "new things" for a promotion.

if you do go over that.

step 5: partition your database and setup sharding and use logical replication to replicate columnar storage on redshift for long term analystics.

whatever is past step 6, you probably wont' get there

If you want to sneak some buzzwords in, work Raft into the protocol so that one machine is serializing all of the transactions but you can replace it with another if it falls over.

Except someone wrote their own and it's buggy.

Fuck raft and distributed consensus. I’ve spent too many nights awake fixing consul.

I think I want to hear that story. I keep very coarse grained data in consul so I haven’t had any problems.

A huge amount of inappropriate technology is used because people have incorrect understanding of what scale is.

I'll stick to my Actor Model thanks. Akka (Scala) and Microsoft Orleans FTW.

Reply on news.ycombinator.com