Hacker News Reader

Comment by AtticusTheGreat | Hacker News Reader

I don't have much ideology behind going with microservices vs. monolith, but what we've done on some recent projects is organize our code into modules that only communicate with each other through a narrow and well defined boundary layer. If we need to split a module out into a separate service, then it isn't nearly as much work to split it out later.

One of the practical issues we've had with microservices that need to interact with each other in real time is ensuring a consistent state across systems. For example, let's say I need to change the status of an object and afterwards, call a separate service to change state there as well. What happens if the call fails in some way? You can't just run all of this inside a single database transaction anymore. Now you have to design your code to deal with several potential failure points and edge cases, which adds complexity. The other consideration is all calls to a service should be idempotent if possible. It makes coding from the client side a lot easier if you can just fire off a call multiple times (in case of local or remote failure) and not have to worry about state.

Just some of my thoughts, since this stuff has been on my plate recently.

View on HN

Transactions certainly make it easier to maintain global consistency, but one possible contradiction with the above is that if your modules are sharing transactions, then your boundaries are no longer narrow and well defined. By definition, your entire database and all its internal workings are now part of the interface.

This is one problem that I've observed with all the monoliths I've worked on. Because modules are colocated and sharing a database is easy, eventually somebody will do it (even if it wasn't originally intended), and you get lots of ostensibly modular code intertwined with other modules in non-obvious and subtly problematic ways.

I've heard this exact argument at work in favor of microservices. I honestly think it's a lazy man's cop out to say it makes sense to insert a network boundary because it's just assumed somebody's going to violate a module boundary. That's a huge tax to pay for being lazy. If you lack discipline it's going to show up no matter how you decide to distribute your complexity. From what I'm seeing the idea that "devops" is going to offset the increased complexity of network boundaries as interfaces is going to be the downfall of a great many of microservice based implementations that simply didn't need to take that burden and risk. I suspect that a hybrid approach is going to end up being the right solution for a great many companies. One or more well factored monoliths with common shared libraries and orthogonal services that each of them use that make sense being a service vs. a shared library.

I take it you've never worked on a monolith project before. There are always reasons (often deadline related, always legitimate, never as a result of laziness or lack of discipline) where developers have been forced to cross module boundaries.

Nobody arrives at microservices unaware of their complexity and complications. Developers are forced to choose the approach between monoliths have their own issues.

Also common shared libraries are a disastrous idea IMHO. They always become riddled with stateful business logic and the difference in requirements between their consumers means they end up brittle, inelegant and full of hacks.

Actually I've worked on a great many monolith projects over the past 20 years, including a J2EE server (WebLogic). And yes that was a huge monolith that had technical debt that needed addressing. When I left there was this future "modularity" project that I'm certain didn't involve introducing network boundaries between the servlet, EJB, JCA containers, etc. to achieve that modularity. And I can tell you that there was a definite effort to enforce interface/package boundaries between the various server components. If you introduced an "illegal" dependency you were going to be talked to about removing it. I want to say that we actually had the build breaking on such infractions after I'd moved to product management.

The point I was making is exactly what you're talking about. That sometimes deadlines force bad architectural choices and that's called technical debt. The laziness and lack of discipline comes with failing to acknowledge and address that debt in the future. As best I can tell people think that microservices are going to solve that problem and I'm saying they won't and that there's not enough thought going into the price of network.

Just like in real life when it comes to being in shape. Diet and exercise. There's no silver bullet there either and it seems to me as though microservices are the fad diet of the current tech cycle.

I realize this stuff isn't cut and dried and easy. If it were then none of us would have well paying jobs to figure out when to use what tool for what job. There's a time and a place for all solutions but I'm seeing the same groupthink I saw back when everyone was purchasing Sun, Oracle, and WebLogic for sites that didn't and would never need those tools. This is EJB all over again as best I can tell.

As far as your shared libraries comment goes, you wouldn't consider having any shared libraries ever? What you're describing are permutations of a shared library that either need to be addressed by the shared library's design by adapting or splitting into multiple libraries. I'd be interested in learning what you do instead of sharing components? Duplicating everywhere?

> Nobody arrives at microservices unaware of their complexity and complications.

I think the opposite is often true. There are developers that buy into microservice architecture without a full understanding of the complexities involved.

> There are always reasons (often deadline related, always legitimate, never as a result of laziness or lack of discipline) where developers have been forced to cross module boundaries.

Those are still a lack of organisational discipline.

Your argument is, in effect: In a modular monolith it is technically feasible to violate the stated architecture. Some organisations will chose to take that option for good reasons, therefore we should make it so it is no longer technically feasible to do so.

I've seen that too. I've done that too.

But let's call it what it is: It's choosing to implement the more complex technical design in order to remove options that you don't want to be available, because if they are available then someone will override your architectural decisions for short-term commercial reasons. And you don't want them to have that choice.

So, either

(a) the organisation is prone to making short term decisions with undesirable long-term consequences, and has collectively decided that they can't trust themselves to stop, and need technical constraints in place.

(b) the organisation is prone to making short term decisions with long-term consequences that the technical team don't like, and the technical team has decided that since they can't convince the organisation to stop, they need to put technical constraints in place.

One more point is that in your monolith you actually have the option of crossing module boundaries. You don't have that luxury with microservices unless you want to introduce XA (God bless you). So you better get your boundaries right. :)

>I honestly think it's a lazy man's cop out to say it makes sense to insert a network boundary because it's just assumed somebody's going to violate a module boundary.

Its not a lazy cop out when the comment he's replying to is the exact example in question.

It helps that network boundary typically becomes responsibility boundary (different teams)... in the end disciplined devs will make it work anywhere, anyhow.

Its a matter of making it work with the cards you have.

I agree about this danger. But then again, nests of services can also grow tangled dependencies, now in the form of RPC calls.

It's a general problem in software: adding a dependency without cleaning up (or even becoming aware of) it's effects on the dep-graph is often the quickest way to solve today's problem. You then pay for it over the rest of the life of the project.

Sure but they are tangled at the macro level.

What happens with monoliths is they tangle at all levels. How many times have you seen a giant "Utils" module which initially contained stateless StringUtils and similar classes then devolved into a dumping ground for stateful business logic.

Or especially with JVM applications how many times do library dependencies for one of part of the codebase end up causing issues with another. That's a big bonus with microservices i.e. being able to manage third-party dependencies better.

For a process that is inherently sequentially dependent on previous results, how is this not a transaction other than declaring it not so or 'dropping the outcome all over the floor' if there's a problem?

It's kind of like saying, "You're not allowed to have this problem, you be better off if you had some other problem like the one I have here"

Unless you're just saying to "grow the boundary" until all parts of the sequentially dependent process is inside the boundary? (this may be tricky to deal with the more external systems there are that cannot be "internalized")

* I am not saying you need distributed transactions - just that some processes cannot easily be encapsulated as "atomic" operations.

We've used microservices for around 6-7 years now. One thing we realized quite early was that letting each microservice store state "silos" independently was a bad idea. You run into the synchronization issue you describe.

Instead, we've moved the state to a central, distributed store that everyone talks to. This allows you to do atomic transactions. Our store also handles fine-grained permissions, so your auth token decides what you're allowed to read and write.

One non-obvious consequence is that some microservices now can be eliminated entirely, because their API was previously entirely about CRUD. Or they can be reduced to a mere policy callback -- for example, let's say the app is a comment system that allows editing your comment, but only within 5 minutes. ACLs cannot express this, so to accomplish this we have the store invoke a callback to the "owner" microservice, which can then accept or reject the change.

Another consequence is that by turning the data store into a first-class service, many APIs can be expressed as data, similar to the command pattern. For example, imagine a job system. Clients request work to be done by creating jobs. This would previously be done by POSTing a job to something like /api/jobs. Instead, in the new scheme a client just creates a job in the data store. Then the job system simply watches the store for new job objects.

Of course, this way of doing things comes with its own challenges. For example, how do you query the data, and how do you enforce schemas? We solved some of these things in a rather ad hoc way that we were not entirely happy with. For example, we didn't have joins, or a schema language.

So about a year ago we went back to the drawing board and started building our next-generation data store, which builds in and codifies a bunch of the patterns we have figured out while using our previous store. It has schemas (optional/gradual typing), joins, permissions, changefeeds and lots of other goodies. It's looking extremely promising, and already forms the foundation of a commercial SaaS product.

This new store will be open source. Please feel free to drop me an email if you're interested in being notified when it's generally available.

and how that central 'data store service' is different than a single 'database service' (rdbms or nosql - CRUD) that all microservices connect to and run there select/insert/update/delete/crud ops?

Other than api - rest vs whatever binary rpc protocol, it sounds very much like a standard database...

The difference may seem subtle, but I'd argue that it is a whole other paradigm. It's one of those things that you either get, or you don't, but it might take some time to fully appreciate.

First of all, we're not an RDBMS, and don't pretend to be. I love the relational model, but there's a long-standing impedance mismatch between it and web apps that I won't go into here. There are clearly pros and cons. Our data store isn't intended as a replacement for classical relational OLTP RDBMS workflows.

If you let all apps share a single RDBMS, you're inevitably going to be tempted to put app-specific stuff in your database. This one app needs a queue-like mechanism, this other app needs some kind of atomic counter support, etc. You may even create completely app-specific tables. How do you compartmentalize anything? How do you prevent different versions of apps to stick to the same strict schema? How do you incrementally upgrade your schemas without taking down all apps? How do you create denormalized changefeeds that encompass the data of all apps? How do you institute systemwide policies like role-based ACLs, without writing a layer in stored-procedures and triggers that everything goes through? Etc. There are tons of things that are difficult to do with SQL, even with stored procedures.

I would argue that if you go down that route, you'll inevitably reinvent the "central data store pattern", but poorly.

Fowler refers to this as the "Integration Database" pattern, and advises against it: https://martinfowler.com/bliki/IntegrationDatabase.html

The issue with a centralized data store is that your services are coupled together by the schemas of the objects that they share with other services. This means you can't refactor the persistence layer of your service without affecting other services.

All that said, a single source of truth does do away with distributed transactions, so I can see the appeal.

He seems to come at it from a slightly different angle, and I can see how his scenario isn't a good idea.

It's worth pointing out that you do have the same challenge in a siloed scenario, but the "bounded contexts" are separated by the applications themselves, which no chance of tight coupling because there's no way to tightly couple anything. In the silo version, apps can still point at each other's data (e.g. reference an ID in another app), there's just no way of guaranteeing that the data is consistent.

The coupling challenge is solved by design -- by avoiding designing yourself into tight couplings.

For example, let's say you desire every object to have an "owner", pointing at the user that "owns" the object. So you define a schema for User, and then every object points to its owner User. But now all apps are tightly coupled together.

In our apps, we typically don't intertwine schemas like that unless there's a clear sense of cross-cutting. An "owner" field would probably point to an object within the app's own schema: A "todoapp.Project" object can point its "owner" field at a "todoapp.User", whereas a "musicapp.PlaylistItem" can point to a "musicapp.User".

(Sometimes you do have clear cross-cutting concerns. An example is a scheduled job to analyze text. The job object contains the ID of the document to analyze. The job object is of type "jobapp.Job". The "document_id" field can point to any object in the store. The job doesn't care what the document is -- all it cares about is that it has fields containing text that can be analyzed. So there's no tight coupling of schemas at all, only of data.)

However... I have played with the idea of a "data interface" concept. Like a Java or Go interface, it would be a type that expresses an abstract thing. So for example, todoapp could define an interface "User" that says it must have a name and an email address. Now in the schema for todoapp.TodoItem you declare the "owner" field as type "User". But it's an interface, not a concrete type. So now we can assign anything that "complies with" the interface. If todoapp.User has "name" and "email", we can assign that to the owner, and if musicapp.User also has "name" and "email" with the right types, it is also compatible. But I can't assign, say, accountingsystem.User because it has "firstName", "lastName" and "email", which are not compatible.

I think the point was not so much about a RDBMS, but from an architectural point of view, you have a central data thingy that similar to a central RDBMS data thingy in that all parts have to point at the central thingy for their data needs. Not so much about the pros and cons of RDBMS

Isolated, internal state is part of the definition of microservices. It is not hard to find articles that assert this. Taking that away may yield a better result, but you no longer have "microservices", you have something else

I don't have a specific definition of "microservices", nor do I think anyone does.

The central data store pattern arguably makes apps even more "micro", albeit at the expense of adding a dependency on the store. But the opposite pattern is to let each microservice have its own datastore, so you already have a dependency there.

It's just moving it out and inverting the API in the process; for many apps, the data store becomes the API. For example, we have an older microservice that manages users, organizations (think Github orgs, but hierarchical) and users' membership in those orgs. It has its own little Postgres database, and every API call is some very basic CRUD operation. We haven't rewritten this app to use our new data store yet, but when we do, the entire app goes away, because it turns out it was just a glorified gateway to SQL. A verb such as "create an organization" or "add member to organization" now becomes a mere data store call that other apps can perform directly, without needing a microservice to go through.

The users/organizations app example sounds like the app is self-contained anyway, I don't see much difference between the app having its own data store and the centralized data store service, it's just which service to call for upstream. What would you gain by moving from the app's own store to the central store service and eliminating the app?

It doesn't just eliminate that app. It generally means no app needs any CRUD except to encapsulate business logic.

Secondly, all data operations can now be expressed using the one, canonical data store API, with its rich support for queries, joins, fine-grained patching, changefeeds, permissions, etc. Every little microservice doesn't need to reinvent its own REST API.

For example: The users/org app has a way to list all users, list all organizations, list all memberships in an organization, etc. Every app needs to provide all the necessary routes into the data in a RESTFul way:

    /organizations              # All orgs
    /organizations/123          # One org
    /organizations/123/members  # Members of one org
    /organizations/123/members?status=pending  # Members of one org that have pending invites
    /users/42                   # One user
    /users/42/organizations     # One user's memberships
    etc.

A client that wants to query this app must first pick which silo to access, then invoke these APIs individually. The way that the querying is done is silo-specific, and the verbs only provide the access patterns the app thinks you want. What if you want to filter invites not just by status, but also by time? Every app must reinvent every permutation of possible access patterns. REST is pretty exhausting that way. GraphQL is a huge improvement, but doesn't really fix the silo problem.

With our new store, a client just invokes:

    /query?q=*[is "orgapp.member" &&
      organization._ref == "123"
      && status == "pending"]

(Yes, we did invent our own query language. We think it was necessary and not too crazy.)

Or indeed:

    /watch?q=*[is "orgapp.organization"]

Now the client gets a stream of new/updated/deleted organizations as they happen.

First, I think when you talk about all the things you've built into the canonical data store (CDS), why can't some of these be decomposed services in their own right? Permissions would be a valuable service to decouple from CDS, for example.

Second, what are the constraints of CDS? How much data can I pack into a single object? silo? How does bad behavior on the part of one caller affect another? What if CDS just doesn't work for a new service you're building?

I do appreciate that your company has invested in providing data storage as a service for yourselves, which I think is a much better idea than having each team rolling their own persistence. However, I think people would be very interested in how you've made sure that CDS isn't a SPOF for all of your data, as well as what kinds of things it isn't good at.

EDIT: I would also point out that there is a difference between having a single CDS and having StorageaaS that vends CDS's.

Those are good and important questions.

Our old "1.0" store architecture did in fact decompose things into multiple services. It has a separate ACL microservice that every microservice had to consult in order to perform permission checks. That was a really bad, stupid bottleneck.

For our new architecture, we decided to move things into a single integrated, opinionated package that's operationally simpler to deploy and run and reason about. It's also highly focused and intended for composition: The permission system, for example, is intentionally kept simple to avoid it blooming into some kind of all-encompassing rule engine; it only cares about data access, and doesn't even have things like IP ACLs or predicate-based conditionals. The idea is that if you need to build something complicated, you would generate ACLs programmatically, and use callbacks to implement policies outside of the store (the "comments only editable for 5 minutes" is an example of this), and maybe someday we'll move the entire permission system into a plugin so you can replace it with something else.

It's also important to note that the store isn't the Data Store To End All Data Stores. It covers a fairly broad range of use cases (documents, entity graphs, configuration, analytics), but it's not ideal for all use cases. There are plenty of use cases where you'll want some kind of SQL database.

> let's say the app is a comment system that allows editing your comment, but only within 5 minutes. ACLs cannot express this, so to accomplish this we have the store invoke a callback to the "owner" microservice, which can then accept or reject the change.

I think these kinds of access control rules, can be expressed within an entitlement solution. These systems are often called RBAC+ABAC (role based access control + attribute based access control). The caller calls a PDP (policy decision point). Policy decision point is a rules engine that can take in the callers application context (which, in your case, will include current time and the time of the initial post) PDP is often implemented as a microservice, or even as a cache-enabled rules engine that, as API resides with the context of every caller (for faster, lower latency, more resilient solution)

These components are part of XACML https://ccskguide.org/policy-decision-points-policy-enforcem...

Congratulations, you've built a microlith!

Idempotency would be nice, but it is often impossible to have at all layers. Eventually at some point, you deal with stateful microservices and distributed transactions. Depending on how long the transactions take, either two-phase commits or compensation transactions are needed to rollback or restore states when failures happen. And that is not trivial to implement and complicates your system further.

Stable and well-defined interfaces between microservices are another luxury hard to have in reality, especially when business and application logics constantly evolve. More often than not, it's inevitable to juggle multiple services to fulfill the need, which takes much more time, effort and risk than monolith.

While there are plenty of stateful services out there (somebody has to store your data or spin up your VM, after all), I think you'd be surprised at how few if any of them require distributed transactions. a lot of problems that appear to require distributed transactions can be solved by more efficient routing as well as more thoughtful approaches to how you approach state.

I would also say its easier to define good, growable, and sustainable APIs when you really think about what the primitives of your service are. Try and avoid baking a lot of opinion into your APIs, which lets your consumers own most of their business and application logic. If you do have a need to embed new business logic into your APIs, think about how you can preserve the current default as well as provide extension points instead of one-off changes.

Hmm, my ideology doesn't say whether microservices are better than "monoliths". But it does roll its eyes when it sees people mistake the encapsulating things into modules for some particular technology helping you do that.

I mean when OOP was new people talked as if (a) no one had been trying to seperate out modules before OOP, and (b) the class was the natural boundary between modules. Both are false.

BTW: what does that article mean about that [in] "Java 9 a native module system is added..."; presumably this is something distinct from the package system it always had. What are the differences?

Author here. Java 9 adds a new module system where module descriptors are introduced to explicitly demarcate the public API of a module, and to express its dependencies on other modules. Example of a module descriptor:

  module mymodule {
    exports mymodule.pkga;
    exports mymodule.pkgb;

    requires someothermodule;
  }

What happens is that every package except the ones exported are accessible to other modules. Non-exported packages are encapsulated, not even reflection can break through that barrier. The requires statements are used by the Java compiler and runtime to verify the current configuration of modules resolves correctly.

Obviously there's lots of more detail to go into. Of course I recommend you check out my upcoming book (early release available) for that: http://shop.oreilly.com/product/0636920049494.do

In short, Java makes a great step forward wrt. modularity. When regular JARs transition to modular JARs (adding a module descriptor), many more checks and balances are in place than are currently possible with the classpath.

>What happens is that every package except the ones exported are accessible to other modules.

I think you meant inaccessible :)

Whoops, you're right!

Java packages are namespaces for code. Java modules are deployable bundles of code. The units of deployment (JAR, WAR, EAR files) pre-Java 9 do not have a consistent model of specifying dependencies and exports needed at runtime.

In other word, packages help the compiler, but give no information to the runtime or deployment as far as how, when, and from where to deploy or load code.

That's a good point, transactions are hard in the micro-service world. In my experience usually It's possible to re-design architecture to encapsulate transaction inside one micro-service. If you have transaction across multiple micro-service are they are really decoupled?

Similar problem is with doing asynchronous requests e.g. using RabbitMQ it is possible only with well designed boundaries as it's hard to control state of request if you do everything asynchronously.

Anyway micro-services is not perfect solution for everything, but even despite these problems I love working with them!

Great observation, and exactly what I meant with the modular alternative in this article. When you can 'get away' with such an architecture, it makes a lot of things simpler.

If you have transactions that have to span multiple modules, I'd argue they are too commingled

One solution is to use a distributed key value store (like etcd) to coordinate tasks. Kubernetes does that (AFAIK); you should definitely check that out.

Reply on news.ycombinator.com