Real World Microservices: When Services Stop Playing Well and Start Getting Real
blog.buoyant.io
blog.buoyant.io
From the React/Flux frontend paradigm, to the "everything is a stream" philosophy of Unix systems, to Erlang actors, to event sourcing for application backends, to databases like Datomic and CouchDB.
This is why I'm so excited about Elm right now - it explicitly puts this at the core of every application, and gives you a really nice language and type system for encoding your domain logic and manipulating data.
Time to write a blog post, I guess.
Event sourcing, on the other hand, is an internal implementation detail of a particular service... Which happens to get you a good stream of events as a side-effect.
The question I really have is how people have managed to upgrade their message bus and to keep availability high with rollback contingencies while upgrading, say, RabbitMQ or Kafka across all your connected services. I really don't hear about that as much as how to do rolling restarts / upgrades of nodes for a service endpoint using some feedback from monitoring and metrics.
"than" and beyond is an illogical statement. Like Y is Z than X.
If what you mean by sharing data is about dependency between calls to different services, e.g. A -> B -> C..., I would go with Storm and other stream processing frameworks, cause that is why those frameworks are built...
> The work described in this thesis is the result of a research program started in 1981 to find better ways of programming Telecom applications. These applications are large programs which despite careful testing will probably contain many errors when the program is put into service. We assume that such programs do contain errors, and investigate methods for building reliable systems despite such errors. The research has resulted in the development of a new programming language (called Erlang), together with a design methodology, and set of libraries for building robust systems (called OTP).
[1]: http://ftp.nsysu.edu.tw/FreeBSD/ports/distfiles/erlang/armst...
I think the answer is that he's doing essentially layer-7 routing, using the HTTP path (and verb?) to decide what backends to route to, rather than doing it on a per-port basis (which is necessary to support non-HTTP services.)
Implementations I've run into before seem to fall into a few categories:
- A service declares that it needs to talk to, say, 3 other services. Each of these upstream services is assigned a unique client-side port, and an ambassador proxy is launched alongside each instance, which exposes those 3 ports, routing each to the corresponding 3 backend services. To keep the ports out of the source code, typically environment variables are automatically assigned, so you just talk to ie. "my-ambassador:${UPSTREAM_PORT_NAME}"
- A service is responsible for using a service discovery layer like zookeeper or etcd to find backends on its own. This is important for things that use raft/paxos/gossip, where blind routing isn't enough, you actually need to keep track of peer instances (although in that case, the service discovery layer is only used for initial discovery).
- Service discovery is done with just plain DNS on well-known ports (or even SRV records if you're lucky enough to have client software that can tolerate them), and you just hope the right thing happens. This can be accomplished with things like SkyDNS on top of etcd... a surprising amount of flexibility can be accomplished by putting logical things right in a hostname.
This approach seems to be like a more opinionated version of the first approach, but instead of ports, it uses HTTP routes, which is definitely more flexible, but only works with HTTP. The routes can contain enough information to route more intelligently than just enumerating your dependency services ahead of time and getting static port assignments.
It's almost like an API gateway distributed as an ambassador to each service. At least, it looks a lot like an API gateway, in that it presents the whole mesh of services underneath in a single URI namespace, using HTTP to route requests accordingly and handling things like TLS and potentially authentication. It's definitely an interesting approach, and something I'm curious to try.
Your explanation sounds a bit "no duh" though, since I can't imagine any way of implementing this pattern that doesn't abstract away knowledge of how to reach remote partners, perform health checks, load balancing, retry logic, etc. It's generally what the term "proxy" implies. Or am I missing something?
Edit: I think my confusion is caused by my experience with running vanilla Mesos (not Marathon or DCOS, but our own custom frameworks), where it's expected that you just have to implement this stuff yourself. I never even thought of open sourcing or productizing what I've written, because I kind of thought that everybody else must be rolling their own quick solutions for these problems too? I admit I'm probably the one with a messed up worldview though.
I think the mesos community is still in the phase of picking the winners for best practices, and hopefully one day it'll be obvious that you use services like linkerd and don't roll your own solution.
I just wanted to point you to the official explanation, since you seemed to be asking for clarification as to Linkerd's purpose.
A proxy can be a lot of things, of course. Linkerd is explicitly designed to route RPC, not just any web traffic. Linkerd is designed to run close to your app and offload all the routing intelligence.
For example, it seems quite common these days to build many types of service discovery and so on into the app itself: App wants to find another microservice, so it looks up the target host in Etcd or ZooKeeper or whatever, then talks to it, handling retrying and load-balancing and so on. If you're using a DNS solution like SkyDNS or Consul, then the app is isolated from the lookup mechanism, but you're still talking directly to your peer. The opposite, older trend is to use something like HAProxy to handle the routing, but HAProxy was designed for a fairly static set of routing targets.
Linkerd is a bit of a middle way: Make the app stupid and put all the operational intelligence in an external process that isn't actually your app, but still sort of behaves like it is. Linkerd is designed to be dynamically configured to support all the kinds of glue you can think of (ZooKeeper, Kubernetes and so on) so that no app changes are needed to support different routing schemes.
I think I'm just jealous that they're getting attention for solving something that I solved too, but didn't even think to release it because I thought everybody else was just like me.
My approach involves running a sidecar that registers a set of etcd watchers, and when upstream services move around, it passes the list of backends through a config file template (using Go's templating syntax), and runs a configurable command.
Meanwhile, other services' sidecars are health checking them, and keeping them in etcd only as they are healthy.
We wire that up to have the watcher process in the downstream sidecar, which rewrites an haproxy config with the list of upstream backends, and triggers an haproxy restart when it changes the config file.
And then, that whole thing is wrapped in a declarative syntax that lets you say "here's the things I want to talk to", and it knows how to find them in etcd, how to construct the haproxy template, and how to restart haproxy, and puts the whole thing in a docker container that links to your container.
And then we wrap all that in a web UI (and CLI) that lets you say "I want my service to talk to that service" and all the things happen for you.
Looking back, it shouldn't surprise me that this level of effort is something people would want an off-the-shelf solution for. But to be fair, I started doing this a few years back so things like DCOS and Marathon and confd and linkerd and namerd didn't exist then. :-D
Curious what people think of a sidecar approach versus the client library approach? Any preference?
The logical evolution of ones architecture when scaling from zero to orders of magnitude beyond is to eventually split out functionality so it can be worked on and scaled independently. That's it, that's all its about. And over time you find this becomes the pattern that helps stabilise and speedup an organisation as the number of people increase.
Most of the time it's an organic process. I would never tell anyone to start with microservices but to merely keep these ideas in the back of their mind and pick tools that simplify the process of moving to distributed systems later.
But right now it's a cool thing, since google is doing it. And Netflix, or whatever. Organisations are about to waste crazy amount of money on microservification of "monolithic" applications that are perfectly fine, because they think it will solve all of their problems.
Microservices are not solving complexity, it is adding complexity in order to solve problems related to scalability and size, and you need to be good to pull it off. If you are working in a half-arsed inhouse development department in a medium sized company in a uninteresting business, chances are that the organisation (as a whole) is not good enough even if there are some smart people around, and you should be grateful that the RDBMS is there and rolls back problems caused by the shitty code that keeps piling up in your git repository.
I started working for about 20 years ago with distributed systems in the telecom industry, where we had to scale telephony services over a lot of machines, both vertically and horizontally. We had key-value datastores, services, application logic, and interfaces as small programs running in it's own machine. And it was bloody hard to handle transactions, routing, fail-overs and generally getting the right balance between calling a service or doing it locally. We did web applications like that too actually - using cgi, a custom html template format and the crappiest script language ever imaginable and it kind of worked too.
It was rather ahead of it's time, much thanks to a few smart guys that were trying to solve the rather hard problem of using cheap hardware for providing very reliable telecom services. It never was "very" reliable, but it was ok. At least on sunny days.
It scaled up to a point. The networks weren't that fast, and the sheer complexity of it all was a limiting factor. We had to create tools for generating the configuration and at one point I found out that we had more than 100.000 lines of configuration in a moderately large system, which explained why it took a while to assemble these systems by hand.
But as time went, the hardware was catching up and you could more or less put everything in one box. And we did. And we started to use RDBMs because we could really need the power of a flexible and transactional database manager, and it was a bliss to write a large monolithic application for user provisioning and administration and not needing to handle every shitty little detail by yourself.
I am very reluctant to going back to key-value stores and really small services, unless we really need the speed. It comes with a heavy cost. (Even though you get more for free now, as more people is doing it.)
The best trade off in my opinion is to build as large systems as possible while it still handles the load and it is still possible to work on efficiently with a few teams. When it's hard to keep up with what is happening and how everything works, it might be better to split the system into smaller systems and (semi) independent teams. The key is the independent team, real devops, and a hard focus on making deployments fast and with no or almost no downtime.
Unfortunately, where I work now, the major limiting factor of our velocity is our dependencies to other teams and systems. If we can build something on our own we can typically do a reasonable feature in between 3 days to 3 weeks. If we are depending on development by another teams changes in other systems, it minimally takes 3 weeks.
And the problem grows exponentially. If 3 systems are involved, it will take us 3 months or more, and failure is always an option.
I don't see how more services will solve that rather hard organisational problem, but that is apparently what some people think will happen, effortlessly, just because technology.
We're breaking down complex code into simple code with complex interactions. Moving the complexity into a few universal platforms that everyone uses, largely in the hope that people smarter than us do that work on the complexity and we can focus on just the little simple bits.
Of course, there's lots that is difficult whilst we collectively learn about what has previously been locked up in organisations like Twitter, Google, etc. How to do a transaction distributed over many microservices? Do RDBMSs still make sense with microservices? (The querying changes from SELECT everything you need, to many SELECT just this bit and so centralised databases see a dramatic increase in simple queries... better suited to key:value stores.)
But yes, exactly what you say... we are.
It talks about anti-patterns that lead to the construction of a "distributed monolith" and how to avoid them.
http://www.microservices.com/ben-christensen-do-not-build-a-...
That's both impressive and not surprising. Orchestration is just really hard. Big systems that are supposed to behave rationally and be easy to upgrade and debug... it seems like such a meager request at first.
It's cool to run your own proxy, however doing that at production level is far from simple, at least with smaller teams.