Ask HN: How do you manage cross-system dependencies? | Hacker News Reader

Ask HN: How do you manage cross-system dependencies?

1 pointperlgeek10 comments

On Debian (and many other Linux distros), managing dependencies on a single system is easy. Just declare that package "web" in Version 2.3 depends on package "backend" in Version 0.3.1 or higher. If you try to install "web-2.3" but your "backend" is too old, apt-get/aptitude will complain.

How do I get the same/similar behavior when the frontend and backend are on different servers?

When I heard of "orchestration" tools, I hoped they'd solve that problem, but the products I looked so far look like schedulers / pipelining systems, not like dependency managers.

What tools do you use to manage dependencies on multiple servers?

(In case it makes a difference, we use mostly Debian Jessie, and deploy our software as .deb packages via GoCD. Puppet is used for configuration management, though not for the deployment)

View on HN

You would architect your application to not be so brittle that deploying one wrong package takes the service down. For example if you have to make breaking changes try to make your backend smart enough to handle web2.2 and web2.3.

Tools like configuration management (chef, puppet, ansible, etc) can help you roll out the different versions , and interrogate servers to see what is installed on them.

So in the case of rpm/deb, if you need to update a package and it comes with dependencies I would let the package manager resolve them. But specifically if your packaging your own application components in this format you would not have control to tell server A to be dependent on server B's dpkg state. That would be complicated and brittle.

You want to use rpm/deb/etc to resolve local dependencies only. For setting up infrastructure that needs supporting infrastructure at a certain version to work well this must coordinated at some level even if by hand.

These days there is more than several handfuls of different ways to approach this. I recommend chef or ansible if your new to configuration management.

Edit: oh you use puppet already I see. Just have puppet update packages as needed, and consider using it for deploys. You can group systems together conceptually and do all kinds of things even blue/green deploys, rolling deploys etc. you can use mcollective to see what packages are where.

Thanks for your reply. It's clear that you've put a lot of thought into it, but I'm still struggling make it useful for me.

> These days there is more than several handfuls of different ways to approach this. I recommend chef or ansible if your new to configuration management.

As far as I can tell, neither ansible nor chef really help me with answering questions like "can I deploy service X in version Y to machine Z, or are dependencies missing?". Or do they? If yes, how? Or did I misunderstood what exactly they should help me with?

Lets give a quasi specific example maybe..

lets say you have service X, is your ApiService 1.2, and it's in production.

Lets say you have machine 1 and machine 2, both running ApiService 1.2

In puppet, you could use hiera to know what version of ApiService a machine should have. You can use mcollective to interrogate these machines to verify that puppet successfully installed ApiService 1.2 on these machines.

Now, to answer the question "can i deploy ApiService 1.3 on machine 1", I think that that would be best answered by testing this in a development/staging environment with another machine that is not serving mission critical traffic.

Here you would do the testing you need, and ensure it works, both in a installation capacity (say ApiService 1.3 needs a newer version of libxml2 or something), and in a general "does it pass QA" standpoint.

It would be your package maintainers job to specify that ApiService 1.3 requires libxml2 version XYZ.

Puppet/etc (or, specifically when dpkg runs via puppet) would fail to install the package if its not going to have libxml2 version XyZ.

Being that it's your package, you would write it in a way that dpkg is able to install it on the target servers your targeting. So, you could even spell it out exactly in puppet to, "before installing ApiService-1.3.deb, install libxml2-2.x728.deb"

If what your getting at, is that if ApiService 1.3 on machine 1, relies on libxml2 being updated on machine 2, then that sounds a bit brittle to me and should be refactored to be more forgiving.

Maybe that helps make it more clear, if in doubt will look for more comments.

Let me see if I got this one straight:

So puppet has some infrastructure for querying package versions (and other facts) (mcollective), and a way to store the dependency data (hiera), but the actual management still has to be done manually (potentially by trial & error in a dev/test/canary environment).

Right?

If yes, that's good to know, but still less than I was hoping for.

Yeah, i look at os-package dependency management as largely a problem that should be left up to the os' package manager I suppose.

So, naturally just like any change, you would want to test the upgrade before rolling to prod.

What are you hoping for I guess? Im genuinely curious. Maybe you want puppet to complain before running that system X doesn't have version C of package K ?

How is that different than it failing on dpkg?

(its possible I'm not understanding..)

> What are you hoping for I guess? Im genuinely curious. Maybe you want puppet to complain before running that system X doesn't have version C of package K ?

In the past months we had several failures where new features weren't working in production, and it was usually due to neglecting to upgrade some components.

So for example one developer creates a new feature in the front end, a second developer creates the backend service, notices a missing feature in a second backend service, and delegates that to a third developer.

When the time comes to roll out the frontend service, the first two developers probably remember that a new version of the first backend service is required, but sometimes forget that there's also a dependency on the new version of the second backend service.

So the feature goes into production, but fails because an indirectly dependent service wasn't upgraded as well.

We only have one testing environment, not one per service, so we don't easily notice the problem in testing.

So, if all the services where installed on machine machine, the developers would just need to declare the proper dependencies (which they can do at the time they discover them), and an "apt-get install <frontend-service>" would resolve the whole dependency chain, and either abort the upgrade if the new version of the second backend service isn't available, or upgrade it too.

What I really want is the same easy of dependency management, even though the packages aren't all installed on the same machine. I simply want to declare all of a packages versioned dependencies, and then say "upgrade service X" and it'll automatically tell me what I need to deploy (or even do it for me).

Currently our workaround is a wiki page in which we document the depenencies, and manually resolve them when the time comes for a deployment. But this is very frustrating, because the problem behind it is solved in any decent package manager -- but only in the single-machine case. So just because we run more than one machine, we have to reinvent the whole dependency management wheel.

And it also frustrates me that I can't seem to find much discussion of this topic, even though I'm quite sure we're not the only ones who have these problems. Am I just looking in the wrong direction?

Thanks for explaining. This is a pain point I think everyone that does service oriented architecture has to some level.

Most shops I have worked in seem to mostly attack this via procedure. Ie - release is OK'd by QA, then deploy ticket is made that specifies what versions go where, and then operations is responsible for executing this procedure and ensuring its sanity.

If you have a lot of hands in the pot, (too many cooks), then this is bound to have issues when everyone is busy and things get overlooked.

There is a lot of automation that can help however. Dependency management in this sense I think has to be more elaborate than just "is X installed or deployed", but you might also want to ask "is X healthy?" Just because a package has been installed doesn't mean its working.

So that is in my opinion why there is no one-size-fits-all solution for this. Many releases are fully compatible with each other, and in simple applications probably even more so, but once you have many teams working on many services, it can be a nightmare.

A lot of places have granular feature management. So, frontend can fail gracefully if feature is not supported on backend. Often backend supports more than 1 fronted version set. You can use versioned api's (like putting a version in an http header, or namespacing features like /api/v1/x, /api/v2/x). (it's been said a huge distributed app like facebook has something like 500 different versions of components all running concurrently)

You can have your deployment automation do things like check to see if backend has required version (which would be something that you would have to put in hiera/equiv of what versions are required), and even more importantly if they are responding to health checks.

One project I currently work on has a very distributed architecture, and every application has to conform to a health check endpoint that returns its state and its version. So we have scripts that do either rolling deploys or green/blue deploys depending on what we are deploying, and we can define the order in which services are deployed. We can deploy them concurrently/rolling, or independent/rolling, or either of those flavors non-rolling, or green/blue.

So if we say we are deploying fronted3.1, backend4.2, and we are replacing frontend3.0 and backend 4.1, this would be seen by the deploy scripts (and operations team) as a single changeset for the environment. So if frontend3.1 needed backend4.1 to function, we would specify a independent rolling deployment, so for each service thats being deployed (starting with the backend), we would roll out server by server at a specified interval, and we would wait for that server to return a healthy health check with the specified version. If this fails the deploy is rolled back and notifications are sent.

^ with this working correctly, we have to have the understanding with the application developers that we will never release a backend that is not backwards compatible with any components that rely on this. (beyond a cutoff version, which those do exist too)

In practice this works quite well, but is not magic, and definitely requires some up front development of automation.

So, sorry for the long post, I think this is a long winded way of saying, my opinion is that this belongs in a configuration management, as the logic encapsulating "what version goes where" gets more complex over time, and so some code, logic, decision making, and failsafes all have to be carefully thought out.

Thanks again for your detailed answer.

Currently I think the simplest way forward for us is to check versions of dependent services in the monitoring health check, so that at least the operators have a simple way to figure out missing dependencies. All other options (blue/green, rolling, ...) require too much infrastructure automation that we don't yet have.

Though it still itches me to write some generic, remote dependency management that's flexible enough to be used in different deployment scenarios and with diverse tools. Maybe later :-)

The closest I've found to something like this is docker-compose though I'm not sure it can be used reliably on clusters. https://nixos.org/disnix/ might be worth looking into though I haven't personally used it.

Thanks, disnix looks really interesting. I'll do some reading (and maybe experimenting).

Reply on news.ycombinator.com