Hacker News Reader

Comment by jrochkind1 | Hacker News Reader

As someone who has not yet adopted to Docker, even though that seems to be what is done on contemporary best-in-class PaaS....

I don't totally understand the maintenance story. On heroku with buildpacks, I don't need to worry about OS-level security patches, patches to anything that was included in the base stack provided by the PaaS, they are responsible for. Which I consider part of the value proposition.

When I switch to using "Docker" containers to specify my environment to the PaaS... if there is a security patch to the OS that I loaded in one of my layers... it's up to me to notice and update my container? Or what? How is this actually handled in practice?

(And I agree with someone else in this thread that this fly.io series of essays on docker is _great_, it's helping me understand docker better whether or not I use fly.io!)

View on HN

Yeah the fly.io stuff is great, beautiful artwork too, top class job.

Re: security updates. It's not handled. There are companies that will scan your infrastructure to figure out what's in your containers and find out of date OS base images.

One thing I'm currently experimenting with is a product for people who would like an experience slightly closer to traditional Linux. The gist is, e.g.

    include "#!gradle -q printConveyorConfig"   // Import Java server
    app {
      deploy.to = "vm-frontend-{lhr,nyc}{1-10}.somecloud.com"
      linux { 
         services.server {
            include "/stdlib/linux/service.conf"
         }
         debian.control.Depends = "postgres (>= 14)"
      }
    }

Then you run "conveyor push" from any OS and it downloads a Linux JVM, minimizes it for your app, bundles it with your JARs, produces a DEB from that, sftps it to the server, installs it using apt, that package integrates the server with systemd for startup/shutdown and dynamic users, it healthchecks the server to ensure it starts up and it can do rolling upgrades/downgrades. And of course the same for Go or NodeJS or whatever other server frameworks you like. So the idea is that if you use a portable runtime you can develop locally on macOS or Windows and not have to deal with Linux VMs, but deployment is transparent.

SystemD can also run containers and "portable services", as well as using cgroups for isolation and sandboxing, so there's no need to use debs specifically. It just means that you can depend on stuff that will get upgraded as part of whole OS system upgrades.

We use this to maintain our own servers and it's quite nice. One of the things that always bugged me about Linux is that deploying servers to it always looks like either "sftp a tarball and then wire things up yourself using vim", or Docker which involves registries and daemons and isolated OS images, but there's nothing in between.

Not sure whether it's worth releasing though. Docker seems so dominant.

> It's not handled.

So... what do actual real people do in practice? I am very confused what people are actually doing here.

LOTS of people seem to have moved to this kind of docker-based deploy for PaaS. They can't all just be... ignoring security patches?

I am very confused that nobody else seems to think the "handle patches story" is a big barrier to moving from heroku-style to docker-style... that everyone else just moves to docker-style? What are they actually doing to deal with patches?

I admit I don't really understand your solution -- I am not a sysadmin! This is why we deploy to heroku and pay them to take care of it! It is confusing to me that none of the other PaaS competitors to heroku -- including fly.io -- seem to think this is something their customers might want...

It seems not that hard to just manually bump your version periodically, right? I get that it's not as nice as having it taken care of for you, but it's not like you're having to manage a whole system by hand

Assume I've never used Docker before, can you tell me what "manually bump your version periodically" means with regard to this question? Can anyone give me concrete instructions for what this looks like in a specific context? Like, say the first Dockerfile suggested in OP?

The Dockerfile has in it:

> ARG RUBY_VERSION=3.2.0

> FROM ruby:$RUBY_VERSION

Which it says gets us "gets us a Linux distribution running Ruby 3.2." (Ubuntu?). The version of that "linux distribution" isn't mentioned in the Dockerfile. How do I "bump" it if there is a security patch effecting it?

Then it has:

> RUN apt-get update -qq && apt-get install -y build-essential libvips && \ ...

Then I just run `fly launch` to launch that Dockerfile on fly.io. How do I "bump" the versions of those apt-get dependencies should they need them?

If you're basing on a lang-specific container like ruby, then it's the version of that container in the FROM line. Notice how ruby images come in various versions of OS (https://hub.docker.com/_/ruby). You can specify that as part of the FROM string. However, they also let you drop the OS part, and only specify ruby version. This will usually default to an image with the latest OS provided by that docker repo. Nothing to bump in this case, just occasionally rebuild your own image from scratch, instead of from cache, to make sure it downloads the latest base image.

This right here. Most of the time you just need to rebuild your image. If the project is being actively developed and built, nothing to worry about. (Unless you pin to a very specific OS version of course).

If it's not, you just need to trigger a build every so often. Maybe this could be a feature PaaS offers in the future.

Docker caches the results of each command, so to "bump" the versions you have to trigger a rebuild of the whole image and tell it to toss the cache. So it'll re-run apt-get update and use whatever the latest stuff is as of the moment of rebuild.

There are a few problems with this, as noted elsewhere in the thread:

1. You have to do an uncached rebuild and repush all of your images, using some ad-hoc company specific process. There's nothing that can do this for you at the end points or service levels, because Docker images are meant to be immutable after build and don't come with the scripts or inputs used to build them.

2. The default is to use caching, so devs may not notice that they didn't refresh their base OS for a while.

3. You don't get notified when updates are available or applied. There's nothing like the unattended-upgrades package that comes with Debian normally, which will apply upgrades and then tell you what happened.

4. Because of (3) the latency is very high. There is no story (other than third party scanners) for getting notified about urgent upgrades. If there's another zero day in OpenSSL then with a standard Linux install you'll get patched as soon as a new package is released and your machines update, so pretty quick (a day or so). With Docker images, it'll get patched on an app-by-app basis if and when people get around to doing an uncached rebuild and repush of the image.

5. Kernel upgrades are a whole can of eels in container-world. People like to think of the container as being a self-contained OS but it isn't. There is a largely unstated and untested assumption that any Linux distro user space can run on any kernel version or configuration, regardless of whether the OS originally shipped in that configuration, and everything will just automatically do something sensible. Mostly this assumption is OK because servers are very simple, but it's not actually guaranteed by anything. A lot of people misunderstand the "stable Linux syscall interface" guarantees and what that means.

It's for reasons like this that I prefer the slightly older way of running real binaries that are exposed to the OS and which use OS specific packages. I configured unattended-upgrades and use LTS versions of the OS, so that security patches just stream in without me doing anything. There are a few downsides to this too:

1. You have to either restart your servers from time to time to force security patches to actually get reloaded into memory, or use the needrestart Debian package - however that only works if you're using package metadata properly.

2. You do need to understand at least a bit of Linux sysadmin. Enough to know how to ssh in as root, use apt-get and so on.

3. The tooling story is poor, hence my musings above about demand for something better. Without Docker, today you're going to be manually copying files to the server, having to learn systemd and how to start/stop/enable services, how to restart them on upgrades etc. That's why I've written something that does it all for you.

I mean, every Dockerfile has a FROM some_base_image line at the beginning, and it refers to an image and an optional tag (which defaults to the `latest` tag if omitted). As far as I know, whenever you build your docker image, it will check for a new version of the base image and download that if needed. So e.g. if you have FROM ubuntu:20.04 at the top of your Dockerfile, and a new version of ubuntu:20.04 is uploaded to dockerhub, the base image will be updated the next time you run docker build, which shouldn't be too long if you're making deployments regularly.

Most places are building software regularly. Once a week update the base layer of the container to the latest OS. Then update your libs and app. Run tests, deploy.

> LOTS of people seem to have moved to this kind of docker-based deploy for PaaS. They can't all just be... ignoring security patches?

They can and they are, IME. I've seen images that literally run some version of Alpine Linux from 3 years ago.

"I admit I don't really understand your solution -- I am not a sysadmin! This is why we deploy to heroku and pay them to take care of it!"

Right. I didn't explain it very well then.

Basically it's a heroku-ish solution but as a tool rather than a service, and where you build locally instead of pushing source code to some remote cloud. You say "here's my build system, go push to these plain Linux VMs". Now, someone needs at least a bit of sysadmin knowledge - you need to know how to obtain Linux VMs, set up access to them, and make sure they're (self-)applying security updates. From time to time you'll need to roll to new OS releases and do restarts for kernel fixes. But that stuff isn't all that hard to learn.

Still - I'd be curious to know where your threshold is for touching Linux. With Heroku you never did, right? Dynos could have run Windows for all you knew? If you had a tool that e.g. you gave your cloud credentials to, and it then spun up N Linux VMs, logged in, configured automatic updates and some basic monitoring then let you push servers straight from your git repo, would you be interested in that? How much did you rely on Heroku support going in and helping you fix app-specific problems live in production?

Same boat here. I'm doubtful that we could take on the additional maintenance work for less than the Heroku premium we pay.

I wouldn't be surprised if I'm wrong and newer tools bridge the gap for a lower price and/or time investment, but I also wouldn't be surprised if I'm right and many places using Docker could save time/money offloading the maintenance to something more like a managed PaaS.

Agreed. I actually have not too much problem with heroku pricing (I could complain about some areas, but it's working for us) -- I'm just worried that heroku seems to be slowly disintegrating through lack of investment, so am worried that there seem to be no other realistic options for that level of service! There don't seem to be other reliable options for managed PaaS that takes care of maintenance in the same way.

I've seen render.com thrown around a bit as an alternative. Haven't tried it out myself though.

There's also netlify, vercel and similar sites but I think they're mainly geared toward all-javascript apps.

For what it's worth, the vast majority of vulns in a web app are in its code or dependencies rather than in the base OS. I haven't actually seen any real-world cases of getting hacked because your docker base OS image was out of date. The only exception I would give would be for like language runtime version, which can occasionally be an attack vector. Switching runtime version usually requires manual testing regardless, so I wouldn't really consider it a docker-only problem.

If you're really concerned, just have a CI job that rebuilds and tests with newer base image versions.

The answer to that is "something that is not Docker handles it". Wherever you're hosting your images there'll be something platform-specific that handles it. Azure, AWS, and GCP all have tools that'll scan and raise a notification. If you want that to trigger an automatic rebuild, the tools are there to do it. Going from there to "I don't need to think about any CVE ever" is a bit more of a policy and opinion question.

I hadn't even thought I would be "hosting my images" on something like Azure, aWS, or GCP to deploy to fly.io. Their examples don't mention that. You just have a Dockerfile in your repo, you deploy to fly.io.

But it sounds like for patch/update management purposes, now I need to add something like that in? Another platform/host to maintain/manage, at possibly additional price, and then we add dealing with the specific mechanisms for scanning/updating too...

Bah. It remains mystifying to me that the current PaaS docker-based "best practices" involve _quite a bit_ more management than heroku. I pay for a PaaS hoping to not do this management! It seems odd to me that the market does not any longer seem to be about providing this service.

You can run something like trivy locally, which will do that particular job. Fly.io might add that later.

It is odd, but the general direction of the market (at least at the top) is becoming less opinionated, which means you need to bring your own. Not sure I like that myself.

The result of a Heroku build pack is also a Docker image. The switching of base layers is a feature of the docker image design, though not normally used by normal docker users.

I think the idea is you have a CI/CD pipeline that will periodically rebuild and redeploy with the latest base images along with your code updates and such.

Reply on news.ycombinator.com