Hacker News Reader

Comment by davidkellis | Hacker News Reader

> You push data consistency concerns out of the database and between service boundaries.

Sing that from the rooftops. That is exactly my observation as well. All the vanilla "track some resource"-style webapps I've worked on were never designed to cope with a consistency boundary that spans across service boundaries. Turning a monolith into distributed services is hard for that reason - you have to redesign your data access to ensure that consistency boundaries don't span across multiple services. If you don't do that, then you have to learn to cope with eventual consistency; in my experience, most people just don't think that way. I know I have trouble with it. Surely I'm not the only one.

View on HN

It's worse than that; it's my observation that most microservice architectures just ignore consistency altogether ("we don't need no stinking transactions!") and blindly follow the happy path.

I've never quite understood why people think that taking software modules and separating them by a slow, unreliable network connection with tedious hand-wired REST processing should somehow make an architecture better. I think it's one of those things that gives the illusion of productivity - "I did all this work, and now I have left-pad-as-a-service running! Look at the little green status light on the cool dashboard we spent the last couple months building!"

Programmers get excited about little happily running services, these are "real" to them. Customers couldn't care less, except that it now takes far longer to implement features that cross multiple services - which, if you've decomposed your services zealously enough, is pretty much all of them.

I've heard of even Unicorns throwing consistency out the window. Apparently Netflix has a bunch of "cleanup jobs" that comb the database for various inconsistencies that inevitably show up.

You can't have consistent microservices without distributed transactions. If a service gets called, and inside that call, it calls 3 others, you need to have a roll back mechanism that handles any of them failing in any order.

If you write to the first service and the second two fail, you need to write a second "undo" call to keep consistent.

Worse, this "undo state" needs to be kept transactionally consistent in case it's your service that dies after the first call.

In reality, nobody does this, so they're always one service crash away from the whole system corrupting the hell out of itself. Since the state is distributed, good luck making everything right again.

Microservices are insane. Nobody that knows database concepts well should go near them

Data consistency is one of those things that sounds like it matters but often doesn't. There's not much in Netflix's platform that screams data consistency is an ultra high priority. Any application that deploys active/active multi-region is by definition going to encounter scenarios where data loss is possible. There's just no way around CAP.

I'd venture a guess that most applications have all sorts of race conditions that could cause data corruption. The fact of the matter is that almost nobody notices or even cares.

A little birdie who worked at Grab once told me that they pretty much don't use transactions anywhere. So... maybe there are problem domains where you can get away without transactions, but I'm quite sure that a marketplace that arranges transportation for fee is not one of them. The next time you're standing in a monsoon waiting for the "coming" car that never comes, remember this post :-)

Aside from that, I've found that even in non-mission-critical scenarios ("it's just porn!") it's incredibly convenient to have a limited number of states the system can be in. It makes debugging easier and reduces the number of edge cases ("why is this null??") you have to handle.

> maybe there are problem domains where you can get away without transactions, but I'm quite sure that a marketplace that arranges transportation for fee is not one of them.

I think you'd be surprised/alarmed at how little transactions actually get used in the software world. Not just on small systems where it doesn't matter but I've seen a complete absence of them in big financial ones handling billions of dollars worth transactions (the real world kind) a day. Some senior, highly paid people even defend this practice for performance reasons because they don't realize the performance cost of implicit transactions. And this is just the in process stuff where transactions are totally feasible, it get's even worse when you look at how much is moved around via csv files to FTP and excel sheets attached to emails. I've spent the last 2 weeks being paid to fix data consistency issues that should never have been issues in the first place.

Maybe when we're teaching database theory we shouldn't start at select/join but at begin transaction/commit/rollback?

Having spent a good amount of time fixing bugs related to code that fetches data with a consistency level of READ_UNCOMMITTED for "performance" reasons I can appreciate what transactions give you as an application developer.

However, I would argue that transactions are overkill. What's the worst case scenario if I book a ride for Grab and my request gets corrupted? I'm guessing I'll see an error message and I'll have to re-request my ride.

I just set everything to level SERIAL and have never seen data corruption because of it. I guess that makes me old school, lol.

Re: I've heard of even Unicorns throwing consistency out the window. Apparently Netflix has a bunch of "cleanup jobs" [to fix "bad" transactions]

If your business model is to be cheap with high volume sales, then corrupting say 1 out of 10,000 customer transactions may be worth it. If you give customers a good price and/or they have no viable alternative, you can live with such hiccups, and the shortcuts/sacrifices may even make the total system cheaper. You are like a veterinarian instead of a doctor: you can take shortcuts and bork up an occasional spleen without getting your pants sued off. But most domains are NOT like that.

> If you write to the first service and the second two fail, you need to write a second "undo" call to keep consistent.

No, you just need loosely-coupled services, where inconsistency in this circumstance doesn't manifest as a problematic end-state.

>It's worse than that; it's my observation that most microservice architectures just ignore consistency altogether ("we don't need no stinking transactions!") and blindly follow the happy path.

If two microservices have to share databases, they shouldn't be microservices.

One microservice should have write access to one database and preferably, all read requests run through that microservice for exactly the reason you mentioned.

>I've never quite understood why people think that taking software modules and separating them by a slow, unreliable network connection with tedious hand-wired REST processing should somehow make an architecture better.

If you're running microservices between regions and communicating with each other outside of the network it is living in, you're probably doing it wrong.

Microservices shouldn't have to incur the cost of going from SF to China and back. If one lives in SF, all should and you can co-locate the entire ecosystem (+1 for "only huge companies with big requirements should do microservices")

>ustomers couldn't care less, except that it now takes far longer to implement features that cross multiple services - which, if you've decomposed your services zealously enough, is pretty much all of them.

Again, that is an example of microservices gone wrong. You'll have the same amount of changes even in a monolith and I'd argue adding new features is safer in microservices (No worries of causing side effects, etc).

I will give you +1 on that anyway because I designed a "microservice" that ended up being 3 microservices because of dumb requirements. It probably could've been a monolith quite happily.

My problem with microservices is the word 'micro'. It should just be services.

Problem domains (along with organizational structures) inherently create natural architectural boundaries... certain bits of data, computation, transactional logic, and programming skill just naturally "clump" together. Microservices ignore this natural order. The main driving architectural principle seems to be "I'm having trouble with my event-driven dynamically-typed metaprogrammed ball-of-mud, so we need more services!".

> Problem domains (along with organizational structures) inherently create natural architectural boundaries

The "natural" order is very often bad for reliability, speed and efficiency. It forces the "critical path" of a requests to jump through a number of different services.

In well built SOA you often find that the problem space is segmented by criticality and failure domains, not by logical function.

It took me longer than it probably should have to realize that the term microservices was made by analogy to the term microkernel. I think that part of my main issue with the "microservices" is that it conflates highly technical semantics with word forms that are a bit more wishy washy in meaning (i.e., a service is something formed more of human perception, not rooted directly in operating system abstractions).

I wonder if microservices will have a similar evolution as microkernels. While some microkernels almost closed the performance gap to monolith kernels, the communication overhead killed them. It turned out that the kernel's complexity could be reduced by moving some functions to the applications ("library OS"). Linux kernel drivers are only accepted when the functionality can't be done in user space. The extreme version is running single purpose, highly specialized unikernels on a hypervisor.

> The extreme version is running single purpose, highly specialized unikernels on a hypervisor.

Which, strangely enough, ends up looking almost the same thing as running services on a microkernel.

Yes! But it’s worse again. Because some bright spark comes along and says “hey there’s some commonality between these n services. I’ll make a shared library that will remove that duplicated code.

Then before you know it there are a dozen more shared libraries and you have the distributed monolith.

Then either have to Stand up every micro service every time integration test or make changes and hope for the best.

When I was designing a microservices architecture for a former employee, the justification for it was security. We needed separation of the components in the system because we were going to be handling money. It used ZeroMQ for communication, because I wanted something more lightweight and falut-resistant for message passing (in such a sensitive application, we didn't want to trust the network). Although it was of course fun to design, microservices weren't my first choice. It just made sense to us to use it in that particular scenario.

Sometimes, I've seen a lack of regard for data consistency within a monolith.

That is not completely on the developer, either. Pre 4.0 Mongodb, for example, does not do transactions. On the other hand, I've seen some pretty flagrant disregard for it just because there are not atomicity guarantees.

Microservices makes reasoning on that harder.

> That is not completely on the developer, either. Pre 4.0 Mongodb, for example, does not do transactions

I'd argue that's on the developer, if he was the one to choose a database that doesn't support transactions, and then didn't implement application-level transactions (which is very hard to do correctly).

Txns are just hard in general but mature RDBMS have spent a huge number of man-years on getting it right. After the time I've spent on Couchbase, Mongo, Elasticsearch, I doubt I'll ever use something non-transactional for anything OLTP or OLTP-adjacent. If Postgres or MySQL can't scale up to handle it, make them...or get a new job. Scale is a PITA.

I've seen it on monoliths too. We run a pretty large app just fine with the "nuclear option". Database transaction isolation level serial. Makes it impossible to have inconsistencies on the database level, and on some RDBMS like PostGres the performance impact is small

They also throw Single Source of Truth out the window way too often.

I remember using BEA Tuxedo 15 years ago and I can't help wondering if theoretically one could do microservices using it or some OLTP to achieve consistency when needed. Has it progressed over the years? Is there any free alternative to it? Or is it dead tech? When I used it it was hard and too much extra work, but it did the job successfully.

Reply on news.ycombinator.com