Dissatisfied with Docker
robert.ocallahan.org
robert.ocallahan.org
Absolutely love Podman. You can even define registries to work with docker hub, easily.
Are there management tools like Portainer available? Or would, even better, Portainer work with Podman?
docker.io provides Debian 10 as far as I can tell.
Not uncommon for rapidly moving upstream. I wouldn't recommend running distro-supplied docker in general (and neither do docker).
Curated apt repos is nicer than snaps, IMNHO.
Tooling would be things like Ansible to setup the container as if you would a normal linux system. Though to be honest, not many people invested in good tooling around LXC, it should be possible without problem to run a docker rootfs in LXC.
With docker, they have the docker.io service with all types of distros you can chose from.
lxc image import metadata.tar.gz rootfs.tar.gz --alias hello
Use them in one command. For example:
lxc launch images:debian/sid/amd64
lxc launch ubuntu:bionic All images available on this server are
generated using community supported, upstream
LXC image templates
Note that the images found on this image server are
unofficial images. Whenever possible, you should try
to use official images from your Linux distribution
of choicedocker.io calls their Debian images "official" and says they are maintained by two Debian developers.
o No primitives to deal with secrets.
o Terrible disk handling (aufs was just horrid, overlay2 I think misses the point. device mapper is just, silly)
o poor speed when downloading and uncompressing images.
Of all of them, the most serious is the lack of secrets handling. Basically you have to use environment variables. Yes, you can use docker compose and stuff appears, but thats only useful if you use compose/swarm.
In this day and age, to have secrets as a very distant afterthought is pretty unforgivable.
I cant see how one would inject data into a process without ptracing/gdb-attaching to it, messing with its internal memory, or just use environment variables or argument list. If you put it on a file somewhere, you still have to tell the process where the file is by an argument or env and also manage that "volume" and uid space of that file and process.
Kubernetes secrets have nothing to do with containers - it anyway usually exposes the secret as env var or volume and file.
Its not even a case of stopping people attaching a debugger, there is simply not standard mechanism. Some sort of secret store should be a primitive in any container/VM/unikernel built since 2012.
The "recommended" way is to map a volume in to a container. But thats rubbish, there is no standard place to put it, and crucially no API to fill the volume full of secrets. How do I map the specific volume to a container? you can't its manual.
what would be nice is either a tagged value store, so you can specify a tag when you build a container, so that you have a way to signal which secrets your container wants. Or a lump of the container that you can encrypt, that has a key value store in it.
There are lots of documents telling you that you _shouldn't_ embed secrets in your container, but there are not many howtos that don't involve the engineering equivalent of gaffer tape and wet string.
Most of this has now been solved by better[] orchestration systems.
[] not great though.
Access control is at best problematic.
Upgrading the daemon without losing state is tricky.
Requiring daemon access to build images is insane.
Building this functionality into something like systemd would be more robust but it's way harder to sell as a product.
That's how I always handle my secrets. (12 factor app)
Am I missing a better way?
You have any examples in mind?
https://docs.docker.com/engine/swarm/secrets/
(as well as the sibling "config")
Everything else is fine for day to day usage IMO (on Windows and Linux at least) and very much worth the trade offs, but having to wait multiple seconds for your app to start is tedious since it plays such a heavy role in both development and even in production. Each second your app is not running is downtime, assuming you're not load balanced.
I opened an issue for this almost a year ago but not too much has come from it other than identifying there was maybe a regression in recent'ish versions: https://github.com/moby/moby/issues/38077
We're talking multiple seconds in Docker vs ~150ms as a regular process to start up a typical web app process like gunicorn in a decently sized Flask app (many thousands of lines of code, lots of dependencies -- a real world app).
Seconds is great in comparison. Minutes, not so much.
Even the hello-world container, which is only a few kB in size needs roughly a second to startup.
You misunderstood. No one calls Docker a “serverless” architecture.
Serverless would be, for example, AWS Lambda, Azure Functions or Google Cloud Functions.
Here is what Cloudflare uses to describe serverless:
> Serverless computing is a method of providing backend services on an as-used basis. Servers are still used, but a company that gets backend services from a serverless vendor is charged based on usage, not a fixed amount of bandwidth or number of servers.
With Fargate you still get charged for your running instances your containers are running on. Even if the containers themselves are idle. This is a container service and not a serverless architecture.
I might be missing something but that seems like serverless pricing. You might be thinking of the pricing scheme when Fargate first launched? Or maybe you’re thinking of ECS, which does in fact charge as you described.
> You pay for the amount of vCPU and memory resources consumed by your containerized applications.
How does vCPU fit into serverless architecture?
> Pricing is based on requested vCPU and memory resources for the Task.
Tasks being a collection of Containers. This is simply a container service like ECS or EKS.
> Pricing is calculated based on the vCPU and memory resources used from the time you start to download your container image (docker pull) until the Amazon ECS Task terminates.
This means you pay for the resources that were started for your containers until the container ends. Including all idle time and any overprovisioning you did because you have to tell it which instance type you want. Compare that to the pricing of Lambda where you only pay for the time your functions need to execute when they get called based on external events.
To bring this back to the beginning of the discussion: Complaining about docker because it is not a good tool for serverless architectures is not smart because it is not used in serverless architecture offerings. Fargate uses containers but it is not a serverless service. Fargate is a container service that tries to simplify setup of compute clusters in comparison to ECS, EKS and EC2.
> Fargate and Lambda are both serverless technologies from AWS.
True, but there's tens of thousands of developers not using big Java enterprise apps where having Docker add 4-5-6+ seconds of startup time has a huge negative impact.
gunicorn (Python) apps tend to start in hundreds of milliseconds (even pretty big ones).
puma (Ruby) apps tend to start in seconds or tens of seconds (it can get higher for massive apps).
cowboy (Elixir) apps tend to start in low seconds (this includes booting up the BEAM VM).
Just the above 3 examples cover the Flask, Django, Rails and Phoenix ecosystem. That's a lot of developers.
Plus there's probably 10x more Java developers than Ruby developers out there. And Elixir/Phoenix devs are probably 0.001% of Java devs :-)
One app I work with has 40 top level gems and about 17,000 lines of Ruby code (which goes a longs ways in Rails). It's not "big big" but it's not tiny. It takes under 10s to boot up. I'm sure if you have a Shopify-tier app it takes much longer, but a majority of apps aren't at that scale.
I was just throwing out examples and I'm just trying to say not everyone is dealing with cases where Docker's added 2-5+ seconds isn't a big deal. For a Flask app those few seconds make it ~20-50x longer to start up due to Docker and for a majority of other frameworks / average app sizes, it's a non-ignorable amount.
- then start your application with docker exec
you will have instant execution.
Only restart your container if the environment changes (new build etc.)
For development, mount your source code in readonly.
Having used rkt in the past, I went to revisit it recently only to find this: https://www.cncf.io/blog/2019/08/16/cncf-archives-the-rkt-pr...
I am so extremely disappointed in the CNCF as rkt (at the time, at least) seemed to be more "production ready" than Docker.
Are there any real alternatives? Is the answer "find something else that uses containerd in a more friendly way?" Is the answer "try to use podman/buildah, which are weird in their own way?"
edit: I didn't notice that you mentioned this in your comment, my bad. Can you explain why it doesn't work for you? I'm new to the Docker-alternative scene and thought it looked pretty good at first glance.
Here's an image straight from a blog post from last year: https://i1.wp.com/blog.docker.com/wp-content/uploads/2018/09...
https://blog.docker.com/2018/09/join-the-beta-for-docker-eng...
Agree the rkt was cool. I just don't know if RH is becoming more like MSFT or more like Oracle.
There used to be an ISP called pilot.net. It was a crappy ISP but to solve its ISP billing problem it wrote a billing system for telcos.
There was a company that tried competing with AWS selling hosting based on hyperthreads of CPUs. It wrote its own provisioning system because it did not want to pay for Virtuozzo. The company's name used to be dotCloud.
Docker is a classic case. Docker must be the craziest, most over-engineered solution for packaging developer artifacts in the Universe.
Brilliant.
I'm going to file this right alongside "software is never finished, only abandoned"
Your quote is first said about art and attributed to Da Vinci.
Sometimes the alternatives are considered superior to the originals (ripgrep) but the originals are just fossilized in place. In 2019 I don't even think we'd be able to have less as the pager instead of more, because we're so conservative about universals defaults, it's sad.
Also the saying is not as pithy if you add more kinds of software.
git clone https://github.com/myprofile/my-cool-app
cd my-cool-app
chmod +x deploy.sh
./deploy.sh
I'm sure containers have valid use cases for larger teams, but for one person I didn't see the point at all - especially as I use the same OS for dev and production.
I find it very ergonomic outside the need to run docker system prune from time to time.
I also value it highly when some open source project I want to play with has docker compose in its readme. Super easy just to spin it up.
- Rollbacks
- Automatic deployment of new releases from CI, rolling releases
- Healthchecks, detecting if/when the server exits and making sure that VM gets killed
- Canary deployments
- Autoscaling (could use an autoscaling instance group for it, but what if you need to scale based on other metrics)
- Log aggregation
- Monitoring
- Service discovery, load balancing between services
- Service mesh
- Fast startup of instances (starting a fresh VM and waiting for it to setup everything from scratch could take ~3-4 minutes, docker is in the ~30s or less range).
- Bin-packing (run a high-cpu low-memory workload colocated with a low-cpu high-memory workload for maximum efficiency)
etc, etc, all of which you get either out of the box or without a huge amount of work if you adopt something like Google Kubernetes Engine (which takes a few clicks to spin up a cluster).
If you don't need any of that stuff and don't want to learn Kubernetes, it's totally justifiable to go that way, but personally I would take Google Kubernetes Engine over something like that any time. There's some up-front cost learning how it works which then pays off very quickly.
bin-packing isn't what you describe, its shoving as much stuff on a machine as possible. Proper resource dependencies management allows what you describe, something k8s is weak on compared to other orchestration systems.
add-job bake-instances --app=my-id --instances=200
Oh, you never wrote tooling and you dont have a jobbing engine? Well, that's the complaiing that you can't get an an Uber because you did not download an app. It is sort of funny because every single shop that is "oh containers!" either creates them manually ( slow and painfully ) or just uses random ones developers find.
> There is a reason containers exist and that every large cloud platform has adopted them, including Microsoft adding support in Windows.
It is the same reason why Whole Foods sells sushi: that's teh way to sell salmon for $85/lb to hipsters. Hipsters hate buying salmon for $22/lb but they love the rolls.
Whatever helps you sleep at night. Creating a container image doesn't have to be slow. There are a lot of tools to help with this that provide things like caching. If I find a container image, the first thing I do is look at the Dockerfile to see if it follows best practices, and if it is secure and lightweight. A lot of the official images on Docker Hub are pretty good. Creating a VM image is going to be slower than Docker period.
> It is the same reason why Whole Foods sells sushi: that's teh way to sell salmon for $85/lb to hipsters. Hipsters hate buying salmon for $22/lb but they love the rolls.
I don't know what this has to do with anything. Besides, I thought hipsters would be vegan.
The second you leave, somebody is gonna rip that crap out and replace it with docker. All while mumbling under their breath...
It's also the approach used by various FAANGs operating at a large scale.
I see a lot of complaints about the docker daemon and root privileges on HN and I've tried to understand where they are coming from but I can't get anywhere. For instance, I understand the reasoning behind "if there is no need for a daemon there shouldn't be a daemon" but I don't really understand what the actual/realized/tangible benefits of not having a daemon would be.
Not using any components with root privileges obviously makes things more secure.
Another benefit of not requiring a daemon or privileges is that then it should be easy to manage fully independent sets of containers. In particular a container should be able to manage its own nested containers in a straightforward way. This is painful in Docker.
I also don’t really care if a container take 2 seconds or 100ms to start... but building docker images is painfully slow.
I’ve also ended up (numerous times) with the “docker daemon is borked” situation, which requires a restart to fix... and you can imagine how that sucks on a prod or multi tenant systems.
-- When our sw gets deployed on-prem by our users (or in their cloud, or wherever), our customers rather not deploy our SW with root for no good reason. Some of them have to do things like fill out forms ahead of time or on use due to this (!).
-- In turn, when we run third-party code, say as a normal dependency or a user-submitted script, we have no reason to trust it. That means we want to limit what it can do, and especially if something breaks out, have defensive layers. (Browsers & NodeJS are Technology From The Future in this regards.)
Note that almost all "docker exploit X FUD" articles, in fine print, make assumptions around running docker as root / privileged / etc. It's 2019, we know better.
Also, we happen to run an increasing amount of our stack as GPU code, which adds a (not so) funny layer to all this.
Here are some advantages of Wasmer vs Docker:
* Much faster startup time
* Smaller containers
* OS independent containers (they can run in Linux, macOS and Windows)
* Chipset independent containers (so they can run anywhere: x86_64, Aarch64/ARM, RISC-V)
I couldn’t read much as I get an “unexpected error” as soon as I start scrolling. It does say that it’s not optimised for mobile.
We will investigate and try to fix it ASAP :)
PS: if you could provide your device info (os and browser) it would be awesome
iPhone XS Max, iOS 13 GM, Safari.
[1]: https://kubernetes.io/docs/concepts/policy/pod-security-poli...
Things I use and love from docker that to me feel "container"-y:
1. OSs as a library (FROM alpine:3.9)
2. Network namespaces so all applications think they are running on a machine with port 80 available
3. Service discovery through DNS
4. CPU and memory share allocation
5. Volumes (bind, temporary, remote)
6. docker-compose for single-command & single-tool development environments (all you need is docker and an internet connection)
Because of that some of the things that you posted are a bit hard to compare.
We believe that by having a VM (based on a industry adopted specification such as WebAssembly) we can control much more granularly both execution (CPU, memory) and the runtime interoperability with the system (networking, file system, ...), solving most of the issues that Docker containers have
Because WebAssembly permits languages to be usable in the web, almost any other language (C, C++, Rust, Go and even Java) is now working to be compiled to WebAssembly.
Therefore you don’t need to adapt your stack to run it in Wasmer :)
This applies no more to the JVM than to Wasmer. There exists WASM to JVM bytecode transpilers [0].
>A container is a standard unit of software that packages up code and all its dependencies so the application runs quickly and reliably from one computing environment to another. A Docker container image is a lightweight, standalone, executable package of software that includes everything needed to run an application: code, runtime, system tools, system libraries and settings.[0]
If you cannot package system tooling, system libraries, and configuration with your software I don't think what you're building is a container system. If you also loose the sorcery benefits docker's network namespaces and volume drivers you're loosing a lot of functionality that docker provides.
I wouldn't call the JVM, RVM, V8/SpiderMonkey, cPython, and any other language VM a "container system" even though many of them allow you to set up platform isolation, restriction, and sandboxing.
I use "container" to reference a technology that allows me to build, package, ship, and orchestrate the execution of these units.
Docker, with all its problems, solves a huge problem for the enterprise. WASM, cool as it is, creates a new one.
I would love to see a WASM + unikernel builder, and some lightweight node scheduler that could plug into kubernetes, so you could run thousands of WASM packages on a node within a k8s cluster, rather than 10s of containers. That increase in leverage would justify the new code stack risk.
- Wasmer isn't available as a package on most distros from what I can tell. Instead they want you to curl a script directly to your shell, which is bad practice.
- They do not have a reproducible benchmark or a test suite to prove that it is actually faster than Docker and it is likely you can't even compare them. The Docker daemon and containers generally have very fast startup times.
- Docker containers are OS independent as well, but rely on Linux kernel features so they run more native on Linux. This isn't a negative. Most microservice apps are Linux-based.
- Adding abstractions for the chipset has always been slower than optimizing and compiling for multiple architectures. This is like someone saying Java is better because it provides a VM that translates everything. Instead Java has a reputation for being clunky and slow because of it. Well built apps and pipelines can easily be compiled for multiple architectures. See Alpine Linux for example.
https://github.com/docker/for-mac/issues?q=is%3Aissue+is%3Ao...
EDIT: It turns out I’m not making this up/an edge case, at least 940 people have run into this too: https://github.com/docker/for-mac/issues/371
Running 'docker system prune' only recovers: Total reclaimed space: 1.986GB
But when I check ~/Library/Containers/com.docker.docker/ for size after prune it still says 64.01 GB.
Am I doing something wrong or is 'docker system prune' useless?
Edit: Before you downvote me too heavily, go give the issues list a thorough read. Some of the problems are quite bad. Perhaps I could have said it in a nicer, but really, it's bad, and it's been bad for a good long while now.
You're always going to get that performance hit with the hypervisor layer there.
Edit: I looked it up. WSL2 continues to use a VM.
https://devblogs.microsoft.com/commandline/announcing-wsl-2/
No, the headline feature of WSL2 is it uses a real Linux kernel under a hypervisor, as opposed to WSL1's approach of a kernel driver running inside the NT kernel which partially emulates the Linux syscall API.
> Edit: I looked it up. WSL2 continues to use a VM.
More accurately, WSL2 starts using a VM, whereas WSL1 didn't. WSL2 runs the Linux kernel inside a VM. WSL1 runs Linux processes under the NT kernel (albeit as a special type of lightweight process called a picoprocess), with an emulation layer to translate Linux syscalls into NT kernel calls.
WSL was a translation layer between Linux and Windows, WSL2 uses Hyper-V (I believe with some special sauce to handle integration).
As I understand it, that what you describe is true of Docker for Mac, but not Docker for Windows which uses Windows' built-in container support (analogous to cgroups).
https://stefanscherer.github.io/how-to-run-lightweight-windo...
That doesn't make the performance problems of docker-for-mac any less nasty though.
Docker for mac is basically dead to me now, going to finish migrating my projects over to the VM and uninstall it from my laptop.
Yeah I don't know why but on Windows the volume performance generally seems to be a lot better than Mac.
I have nothing but good things to say even on 5 year old hardware. Massive web apps (including using webpack) are very very speedy with volumes with a number of different stacks (flask, rails, phoenix, webpack, node).
I still use that set up years later and it's really solid.
The cgroups and Linux specific hooks keep Docker from being implemented natively anywhere else. The fact you have to share the entire Docker socket for containers to be able to control other containers, or that it's not trivial to run Docker-in-Docker, is terrible.
I did a similar writeup about the things I hate about container orchestration systems:
https://penguindreams.org/blog/my-love-hate-relationship-wit...
FWIW, if you enable the remote API, which, granted, isn't as trivial to do securely as it should be[1], then you can connect from any Docker client by simply setting the `DOCKER_HOST` env var and using the right TLS certs. This makes Docker-in-Docker much easier to manage, avoids the security issues of sharing the Unix socket, and works remotely of course.
https://github.com/sumdog/bee2/blob/master/ansible/roles/doc...
It creates client certs and copies them locally too, so I can connect to Docker remotely over a VPN. Still this doesn't solve the original problem I talked about. It's not about securely connecting to the daemon. Even if you connect securely, you still essentially have root access on the host machine.
I've considered writing a proxy that restricts what commands can be forwarded on to the Docker host socket (e.g. allowing for container IPs x,y and z to restart containers, but not to create new ones or pull images). There doesn't seem to be fine grained security or roles built into the docker daemon itself.
Running docker in a docker container would give you a throw-away docker to use for things like Jenkins, Gitlab-CI, and other build tools without giving it access directly to root on the host.
Your original point was about the pain of sharing the Unix socket to control other containers, so that's why I brought up the API approach.
It's been a while since I used Docker, but have you tried enabling user namespace remapping[1]? I remember it working as documented, and don't see why it wouldn't work remotely or DiD. There's also experimental rootless support since 19.03[2], maybe give that a try. Other than that, make sure you trust the images you run, or preferably, inspect every Dockerfile, ensure that the process runs as an unprivileged user, and build everything from scratch yourself.
I agree with you that this is a major security issue, but we've known that since its introduction, and things seem to be improving, albeit slowly.
Thankfully, nowadays there is other OCI-compatible tooling you can use and sidestep Docker altogether. Podman[3] is growing on me, mostly because of rootless support, though it's not without its issues and limitations.
[1]: https://docs.docker.com/engine/security/userns-remap/
[2]: https://github.com/moby/moby/blob/master/docs/rootless.md
[3]: https://podman.io/
Docker provides a nice high-level layer for the end user on the developer machine. Its CLI is efficient, so are the builds when they're local, with a cache. I still use it on my dev machine despite being aware of the alternatives.
Anyone security-aware company using containers in production most likely go with non-Docker approaches. Some notable examples include runc, Podman, Buildah, img, ko, Bazel/distroless.
You can build images using Bazel with distroless as the base image. Similarly I think Jib/ko use distroless images without needing a docker engine.
Still, Docker is the best method that I've seen for distributing software, especially cross-platform. Not just for shipping a containerized web app to production, but also running dev tools (e.g. linters, compilers) and other desktop and CLI applications. I know some people run lots of stuff in containers (https://github.com/jessfraz/dockerfiles, probably the most prolific), but I think this is a largely underappreciated facet of Docker.
My team at work is heavily Mac while I'm on Linux. The dev workflows are identical on my machine, on their machines, and in Jenkins. Nobody has to have a snowflake setup, installing this or that version of Python, we're all just `docker run`ning the same image. It's great.
Unfortunately, Docker for Mac's IO performance is abysmal. If past performance is any indication of future results, that's never going to change. I'm constantly on the lookout for other ways to share tools cross-platform that don't involve Docker. Things like podman and binctr are exciting, and I've played with them, but I don't see them filling this niche.
Are they running the same OS? The same architecture? Do they have any software installed?
start/stop is merely a state change.
1. extract all kernel state related to the target process
2. replicate the state on another machine
Both of these are extremely complicated problems.
The first problem is difficult because it’s nearly impossible to isolate all relevant state in the kernel. Process state is closely intertwined with the entire kernel state, making it all too easy to produce an incomplete snapshot that can break the system in all sorts of ways. Worse, things would get more complicated if multiple processes are involved.
The second problem is just as hard if you consider external dependencies or different kernel versions. It might simply be impossible to recreate the process even if you did succeed in creating the perfect snapshot.
Process snapshots are simply not worth the trouble.
Does podman have better startup performance than docker?
Using a docker to debug and it is a bit of a pain.
* Does not give or require root.
* Fast: Does not write to disk.
* Fast: Does not allocate memory.
* No daemon.
Interesting philosophy, it will pose some issues when replicating a Docker 'build' like system though. But that could be separated from the 'run' system with cached layers.
edit: Btw, Docker also seems to support this: https://nickjanetakis.com/blog/docker-tip-55-creating-read-o.... Also allowing explicit creation of writable tmpfs mountpoints.
As a developer I'd like to be able to check out a docker image, work _in_ it, and merge it to master when I'm done. When my code is deployed, I'll know _exactly_ what is running.
Managing secrets and environments (so the container knows when it is in production instead of running on a developer's laptop) is important to get this to work well.
It feels like it is half way there already. I'm looking forward to when it is as straightforward as git for a developer to use. I'm not too worried about start up time - the biggest drawback to slow startup time is when you are running very sensitive autoscaling that is tearing down and spinning up new nodes very quickly. If you have that problem you may want to rethink your node size, hysteresis, and scaling thresholds.
Dockerless: https://mkdev.me/en/posts/dockerless-part-1-which-tools-to-r... https://dzone.com/articles/dockerless-part-1-which-tools-to-...
What's funny is, Docker is only actually useful because of how many features it has, all its supported platforms, all its bloat. You won't ever be totally satisfied with any alternative because it takes so long to make something like Docker, and someone will always need that one extra feature.
You will need a daemon running as root to bind to ports below 1024.
In addition, in many cases you want a bind-mount onto your filesystem that also supports arbitrary UID/GID on files, which means you will need a root daemon. The problem is of course that anyone having access to the docker daemon can simply say "bind host / to container /mnt" and then can hijack /etc/sudoers for a privilege escalation on the host.
It's mutually exclusive to have usable containers and a system that is secure against privilege escalation by the users at least and (in case of docker-in-docker implemented by bind-mounting the Docker socket in the container) by anyone accessing the container and achieving RCE there.