Comment by lobster_johnson | Hacker News Reader

Comment by lobster_johnson | Hacker News Reader

lobster_johnson

The article misses an important aspect of microservices: They're reusable!

Almost all of the swathe of microservices we've developed internally are general-purpose. We've built a dozen or more user-facing apps on top of them. If I wanted to build a new app today, I would typically sit down and write a Node + React app with no backend code needed because I can just call our existing services.

For example, we have a microservice dedicated to storing documents. With this I can create a todo list, a blog app, a Reddit-type link aggregator with comments, etc. If I need login, there's a microservices that mediates between an identity data model and an OAuth account registry. If I need an upvote/downvote system, we have a microservices optimized just for that. We have services for sending notifications across different transports (email, SMS); "followings" and sending digests about updates to things you're following; organization trees; verifying email addresses and mobile phone numbers and other verification sources; processing photos, audio and video; collecting events for aggregating in an analytics store; etc.

This ability to "pick and mix" functionality you need is the real beauty of microservices, in my opinion. It's a huge time saver. We just whipped up a new site recently where 95% of the work was purely on the UI, since all the backend parts already existed; the remaining 5% was just code to get data I to the system from a third-party source.

This does require that you plan every microservices to be flexible and multitenant from day one. It's a challenge, but not a big one.

Out of curiosity, what management tool do you use to orchestrate all these microservices?

Right now, a combination of Puppet and a simple custom-built tool written in Ruby. The tool inspects the metadata from Puppet about which nodes should run what apps, and the tool uses SSH connections to clone from Git, write config files, restart daemons, etc.

At the same time, developers run a local Vagrant VM that is configured identically to the production/staging clusters, and can run every single app (controlled using the same deployment tool). We also made it simple to mount an app you're working on, with code hotloading automatically taken care of. If you want to work on a microservice or an app, you just modify the code on your local machine and reload the page (or do an API request).

It works really well. It's not perfect, though. Since we're deploying directly to the servers, every node has to install packages, compile modules, package assets, etc., and everything has to wait on the slowest node; we'd like to transition to a system where we build a "release tarball" (or even a Docker image) once, and then push it to the cluster. Also, we're looking into using something like Mesos to better orchestrate apps and move away from role-based nodes; Puppet is way too rigid and too "opsy"; we'd like to use Puppet for controlling the base OS, and let the microservice/apps world be more dynamic.

Makes sense! Have you considered Ansible?

Ansible doesn't give us any benefits over our current system. (Salt would be a better choice. Not a fan of YAML templating, to be honest, nor the tight integration with Python.)

Configuration management systems like Ansible and Salt and Puppet are fairly rigid; they are basically modular recipes for placing files and starting services on remote servers. What you get from Mesos (combined with its frameworks) is something that can run your services, keep them running, and handle the several state transitions that affect any running application: Transitioning from one version to another, for example, or restarting a failing app.

We've tried using supervisord as a stopgap solution, but it, frankly, sucks. Doesn't follow forks (no cgroups), isn't capable of cleaning reloading its config, can't do master/worker replacement, doesn't support syslog in any meaningful way.

all those help with the infrastructure part, take a look at http://github.com/mashape/kong for helping with the code.

I don't know Kong very well, but having look at the documentation, feels a little too much like a framework to me.

The nice thing about Mesos etc. is that it makes almost no assumptions about what your system is. You pick and mix the functionality you need. For example, Marathon is basically just a process manager. You wire it up the way you want; the application can be anything, from a web server to a one-off script.

Kong seems to be "Rails for microservices". Which is probably useful to someone, of course.

I'm curious, what does your document storage microservice look like?

It's a very simple layer on top of PostgreSQL that stores basic JSON documents. Documents are organized into paths, which can be queried, and it supports things like tags and putting documents into timelines, and it does path/document-level permissions.

It does a great job for basic document-oriented content storage, and we use it for many things, but it admittedly falls down for anything requiring inter-relationships between documents (ie., graphs), and it has some other issues. It doesn't support querying on arbitrary fields, so we rely on indexing all documents in ElasticSearch, which introduces a lag problem where the UI needs to explicitly wait until ES has been populated after a change. Also, it's written in Ruby and uses HTTP, so it's not fast.

We're working on a "2.0" that changes it completely. It implements operational transforms/fine-grained revisions, supports multiple backends (such as Redis), fast synchronous updates to other data stores such as ElasticSearch, and uses a faster RPC protocol. Currently writing it in Node.js/ES6, but might rewrite in Go once we see that the design works. It will be open source, if you're interested.

Thanks for the explanation. Sounds like a really cool project.

Reply on news.ycombinator.com