Lessons Learned While Building Microservices | Hacker News Reader

Lessons Learned While Building Microservices

blog.runnable.com

60 pointssundip25 comments

One of the stated lessons learned is: we'd still be on node 0.10 if we weren't using microservices. I've worked a lot on huge monoliths and microservices. Being "stuck" on a version has less to do with monoliths/microservices than it does consistent update practice. If you are using FOSS and schedule periodic updates of the core language and the apps dependencies you'll prevent yourself from getting stuck on old versions. If you run across some dependency that isn't working on version X of the core language you can open tickets against it and/or schedule updating the dependency yourself (this doesn't go away with microservices). Opening tickets against dependencies as the core language updates helps both you and the dependency owner by helping them test and get out ahead. Not only will you prevent from getting stuck on old versions but you'll also have the latest security updates. Note: you don't have to be on the bleeding edge but you should strive to be, in my opinion, no more than 6 months or so from the edge.

> If you run across some dependency that isn't working on version X of the core language you can open tickets against it and/or schedule updating the dependency yourself (this doesn't go away with microservices).

What does go away with microservices (or, in my case, a suite of applications since my stuff doesn't run online), is that each portion only deals with its dependencies. If I have two applications or services X and Y. X depends on D depends on language/library version N. Y has no such dependency, Y can migrate to N+1 while the issues with D are worked out (changing the dependency, fixing the depency, removing the dependency).

In a monolithic application, this is more difficult or impossible. You gain a lot with a monolith, but you lose the flexibility to maintain components as individual components.

So while I'm not anti-microservices, from the developer's perspective sometimes there's just way too many moving parts.

one code change+one test change back in the "monolith" days (and if the change broke the existing tests you know right away) now becomes an ordeal.

Tracking down the client services/altering their code + tests+ simulators (not to mention that you now have to check the code out from bazillions of repos, run it locally etc) and then waiting for their respective build plans to pass, wait for the integration tests to pass etc.

I've done a lot of work on monoliths, and a fair amount with microservices. I've never observed the perceived simplicity of the former to bear out in practice; there's just as much complexity, but instead of being reasonably well contained and prevented from spilling across service boundaries, it's smeared all over the application, so being confident you've tracked down and addressed all of it is much more difficult than it has to be.

It sounds like you may be having problems not because the architecture is designed the way it is per se, but because it's poorly documented and hard to discover. There's nothing about the monolithic style which prevents that, or even makes it less likely to occur.

As I see it (and I haven't been convinced otherwise), the only advantage microservices have over monoliths is that they can scale horizontally more easily, and a slight advantage is that they pressure you to write cleaner interfaces.

There's nothing preventing a monolith from being as modular as a microservices architecture, only without all the headaches that come with monitoring tens of services and sticking a network between them. There is NOT just as much complexity in a monolith, because you have no network of services to monitor, there's only one service.

Sure, your business logic is going to be the same in either approach, but you're comparing badly written monoliths to well-written microservices. There's nothing that says microservices can't be a mess (and I've seen multiple people completely botch it and have all the services talk to the same database).

Well, I've seen a lot of badly written monoliths. Whether or not that's coincidental is a point worth considering.

Polyglot can be nice too.

Ah, true, that's a nice benefit, agreed.

Microservces enforce some separation of concerns, but Monoliths can use the same approach without nearly as much complexity. Overall, I find microservices to add a huge range of new failure modes for little net gain. The sweet spot seems to be services where large chunks of the application operate independently. 10 services are manageable, 100 are not.

> Monoliths can use the same approach without nearly as much complexity

Do they?

> The sweet spot seems to be services where large chunks of the application operate independently.

Which could be said to be a design choice in many cases, perhaps even most.

> Do they?

Ideally OOP should treat each object like a microservice with an interface, independent data store, and test suite. I have seen several applications that do this, but a big ball of mud is also common, so it depends on the team.

PS: IMO, I think something is lost when languages don't have both objects and records. Passing around structured data can make for a clean design, but people get stuck on the idea that data and computation must be linked which makes it harder to have clean separation between different parts of your application. Function X is used by A, and B so now you can't change X without impacting A and B. Wait if we make a sub class then...

Ideally a lot of things would be other than they are. There's a cogent argument to be made that the microservice architecture offers value in making a big ball of mud harder to create. Of course, there's also a pertinent question to be asked about whether one big ball of mud is really that much worse than several smaller ones flying in loose formation.

"It depends on the team" is probably about as close as it's possible to come to a single right answer here.

PS: I absolutely agree.

> it's smeared all over the application

How about writing libraries with clean API and encapsulation ? because basically your argument is "Our developers aren't disciplined enough to write clean API, so we need to force it with http calls ...".

If discipline were the sole answer required for every problem in programming, many things would not need to exist.

Microservices are a great idea in principle. In practice, they require you to spend way too much effort (IMO) on:

(0) Serializing and deserializing objects. (At least if you're not using something like Erlang.)

(1) Validating things that would be considered internal invariants under a more monolithic design, simply because they lie on a service boundary.

Not to mention microservices constrain your API to what your transport (often HTTP) can readily express (which is often not much). If you're used to working with expressive languages, the expressiveness drop can be very annoying.

I've never run across a microservice architecture that involved passing objects across service boundaries. Data records, yes, but if those are expensive or painful to serialize or deserialize, I'd say that's the first problem to solve.

If your data structures have sophisticated invariants, serializing and deserializing them will be painful no matter what, because every invariant becomes one more thing to check after deserializing. And, even if your data structures don't have any invariants, don't you think it would be nice not to have to manually serialize anything? At least it would be nice if the language did this for you automatically.

well, so here's exactly what I worked on just the other day and how it went.

Step 1 - there's a user story that is telling me that one of the services needs to have ONE new data field added to the data model of the domain object it is in charge of. The actual data for that new field will be sent in by the existing services which now will optionally provide the data for the field in the payload. Simple right?

Step 2 - I check that data service out from the repo and start hacking. After about 45 minutes I have the change made locally, have the functional tests added (copypaste existing ones/rename them/add my new field in there), can hit the service endpoint with altered POST and PATCH payloads and GET gives me back what I need.

In the monolith days, my next step would be to alter the tests that will drive the service I worked on to add the data field I added, run complete functional test suite, once everything passes/builds - deploy locally, hit it with integration tests, fix whatever breaks, check into repo, wait for the CI flow to be done and I'm DONE.

But. We're in the microservices world. So I'm now on to Step 3: Step 3a - identify "data provider" services, pull them down from the repo, import into my IDE. Alter the simulators representing the service I changed to have the change I introduced reflected in there.

Step 3b. Alter the services themselves, run tests against the simulators.

Step 3c. Fun part - messing with newly altered client service config settings locally so that they'll simulate every external call except when hitting my altered service's endpoint. This uncovers some unpleasant dependencies like I now need to pull down a docker image with the MQ provider 2 of these services won't start without. Image pukes on startup, takes time to figure out, but now I'm on to the next step.

Step 3d. Now that I can hit my service from the client services I run the tests. Separate test suites from separate projects so I need to switch between terminal windows, IDE projects etc. I mean I'm not digging trenches but my command-tabbing is getting old so I go to lunch :)

Step 3e. Integration tests. What a pain in the ass. Too much to type to describe the ordeal :)

Step 3f. Checking in changes for 4 projects, so basically CIx4 compared to monolith. Now I also have to make sure my changes will get merged into the branches that drive CI with integration tests in proper order, otherwise the CI job will fail on the integration test step. Of course, re-running the build will fix the problem after the "backend" changes are there but still. And plz don't tell me to wait for my pr for the backend service change to be merged and CI completed before checking the client ones in :)

Good thing I didn't have to mess with any configs, that would be even more fun.

your comment reminds me of this tweet

https://twitter.com/honest_update/status/651897353889259520

  We replaced our monolith with micro services so that every outage could be more like a murder mystery.

My current company learned this a couple of years ago - on their first Node project after deciding to move away from .Net, it ended up a giant monolith, and so 4 years ago or so they decided to move towards microservices. The burden of trying to upgrade a monolith is colossal, and it is where microservices shine.

Yup, and even better when you aren't even tied to any language.

Does anyone know of a good resource for learning microservices best practices? We're at a stage at my company where it makes sense to consider the transition from a monolith -- but would love to do the right homework first to avoid some of the pitfalls.

I highly recommend reading Martin Fowler.

http://martinfowler.com/articles/microservices.html http://martinfowler.com/eaaDev/EventSourcing.html

If you prefer books, than I can recommend "Building Microservices" from Sam Newman [1]. You get some basic information and pitfalls you should avoid when building microservices.

[1] http://shop.oreilly.com/product/0636920033158.do

We put together a number of talks on microservices.com/talks that you can check out.

This one from Uber about their scaling is a good place to start: https://www.microservices.com/talks/scaling-uber-from-1-to-1...

And this one from Ben Christensen of Netflix and Facebook talks about some of the pitfalls https://www.microservices.com/talks/dont-build-a-distributed...

Reply on news.ycombinator.com