Gocker: Docker implemented in 1.3k lines of Go
unixism.net
unixism.net
> Or Gocker, an implementation of Docker in go...
With the advent of search engines I discovered that any story I can come up with actually happened somewhere in the world and has an article about it already.
Example: Fake barn county. You're driving in a county and see lots of barns. You point to one and say "that's a barn". Unbeknownst to you, almost all the barns are not barns but are facades. By accident you actually are pointing at one of the few real barns.
[0] https://github.com/moby/moby/commit/a27b4b8cb8e838d03a99b6d2...
That said (and I've said this before), this is not really Docker. It's running containers, not the same thing. If you want to compare it to anything, it's runc, but that's not a good headline :)
Not that the docker architecture is that clean, but it is the combination of ideas it brought to the table what made docker docker:
- have APIs to do everything, from launching workloads to building images. - combining layered filesystems with os-level namespaces - package format for "images" coupled with a distribution system
If it was just starting containers, that was already possible for a good while (and many shared hosting providers already did this).
I think many people underestimate the importance of the first point, having an API to do all this. It's having this combination of ideas that democratised cloud computing, it is what makes the bigger picture possible. While the Docker API is currently not very important anymore, it showed the possibilities, and made the limitations it had also very apparent, and at the beginning, nobody had solutions for this. It took things like Mesos and kubernetes to take it to the next level, with the latter having become the de-facto standard container API.
I don’t know, gunc is a pretty good name.
That being said, the code here is pretty approachable and they weren't noticeably trying to cram it into fewer lines. Like the library, not this marketing.
I recently worked with a client on an integration effort that had to touch many different points on a (massive, for me) codebase with tens of millions of LoC. For that, LoC was the only reasonable metric I could come up with to try and convey the scale and complexity of the task at hand--being quite ignorant of the system's (and subsystems') architecture(s) at the time.
That was further complicated by the way the massive codebase supported all sorts of dynamic compositions and certain interactions needed to work with baseline compositions expressed in a form of markup totaling about about 5 times the amount of the actual code base, amongst other things.
These folks thought the integration could be done for $80k tops and in a couple months. It took LoC metrics to get them to understand the potential complexity at hand and that a lot more time needed to be spent in assessment and design before jumping in.
What really bugs me about LoC as a metric though is when people don't count libraries. "I made a 3D engine in 10 lines" is just a flat out lie if one of those lines is '#include "Unreal/Engine'.
> What Programming Language Does Docker Use? Docker is written in the Google Go (golang) programming language. To learn why Go was used, we’ll refer you directly to Google.[1]
[1] https://blog.stoneriverelearning.com/docker-101-what-is-dock...
Sir we need you to come with us.
But that misses the point. This is very helpful for understanding how a Docker-like system works since it's a small and mostly self-contained implementation. You can read through the entire thing and fully understand it.
It only cheats in the container registry handling where it pulls in github.com/google/go-containerregistry and for the network setup where it uses github.com/vishvananda/netlink. The rest is done in terms of Go stdlib and syscalls.
Early Docker used external utilities (lxc, iptables) and had the client/server stuff already so it's not as straightforward.
I wonder if that can run in unprivileged docker.
FROM ubuntu:focal
RUN apt-get -y install libssl-dev
<your app details>
Of course, you can refer to a specific container image id and pin all your packages, which would go a long way to improving reproducibility.
But once you start getting time differential and caching, you run into... if not bugs, faults - your cached image came from a month ago, apt is now invalid. Sure, I should've used a private dpkg repository, but I want the side-effects from all of those.
A docker image I built today, is very likely to be ostensibly the same as the one I build in five minutes, but it doesn't guarantee that because its cache is at the image-level and the externalities (see: every possible step that isn't an ADD or COPY) is variadic.
It does a very valiant attempt at consistency to be clear, and I have my own gripes about it, but it's not like it creates a new planet from which technology can be built upon; It's more like creating a moon on which you can build a base, but you're still subject to the orbit of the earth.
Does the tech guarantee reproducibility if you can use it to create un-reproducible artifacts? I don't think Docker claims anywhere to guarantee reproducible builds...
But an app built in a docker container is not guaranteed to have reproducible builds, because docker doesn't say anything about what happens beyond loading the initial image
If you use a local clone/submodule of nixpkgs (the git repository with the definition of all nixos packages) at a specific commit, then you will always install the exact same software, because definitions in nixpkgs all specify the exact version and the hash of all the inputs.
- Step 3: pip update && install X
- Step 4: run step-tool
- Step 5: pip update && install Y <build failure>
To be clear, I'm sure all these things can be solved by a complex enough stream of shell commands, but I'm also forced to shove in updates at every step of my build, which is an artifact of the build-system
It's not the equivalent of handing over a VM image. You're still open to unintended sensitivity to kernel versions, for instance.
* A Docker image normally contains all the dependencies of a program, or a set of programs. You can run libraries and other software of whatever versions, not necessarily available on your host system; they are already baked into the image. Usually a Docker image only needs a compatible kernel (this is a very lax restriction). It is a damn easy way to distribute software, especially such software which is not trivial to install and set up: tired of wrangling with Grafana installation? just pick a container from their site. And of course you can mount whatever you need inside the container when you need to, so it has controlled access to your filesystem(s).
* A Docker image normally runs with its own firewall. That is, you explicitly say which IPs / ports are available ("exposed") from the container; everything else is blocked. This helps isolate containers from the internet and from one another, and also helps build private networks between containers not exposed outside the host machine. Since containers already talk to each other via a network, it becomes easy to distribute them across many machines when you need to scale.
* A Docker image is built out of layers, and they can share layers. If you are reasonable enough to put common stuff to the bottom layers, then you can have multiple containers with a lot of common software installed inside (like a Node runtime, a JVM, etc) which stored on the host system only once.
* Docker images / containers are the standard for many cloud management systems. AWS can run containers directly. K8s operates on containers. Docker itself offers a simple but rather reasonable orchestration tool called docker-compose. It's great for small deployments and for things like running your setup locally, for development and integration testing.
Containers are not always better for everything you can think of. But they solve a number of common problems; some of these problems might be ones you'd like to have solved, some not.
- You need persistent storage? You better define it or you'll lose it one the next (re)deploy. - You need to expose network services? Tell me which-ones or it won't work.
If we're talking on small scale single server deploys, it makes backup, upgrade/rollback and migration of applications a LOT easier.
As long as you're not talking about a k8s cluster - which you should avoid with application architectures that aren't "cloud native", I assume you'll have a local docker-compose file which you just start/stop to bring the entire application stack you need (database/app/proxy server/monitoring/...) with one command, which means all external service dependencies are also contained in one 'stack'.
What I also use it for on small-scale apps is having a test environment of the same software running on the same droplet. I just put a Traefik reverse proxy in front of it that autodetects the docker containers, handles HTTPS/ACME certificates and routes the test-url to the test-containers, the real URL to the "production" containers, and they're all isolated.
xinylines.io - implemented in zero lines of code, and one large shameful html file.
Apparently the first version of Jails was a few hundred lines of code.
The dependency on netlink adds a little to the code weight. Some of this also feels like it could just be a shell script (sh, unlike Go, ships with built in Linux support for netlink, and sticks with dotted-quad-string types for IP addresses instead of mixing with int32!)
I did not realize cgroups were this simple to manipulate. Thank you for the enlightenment.
ping 0x7f000001
http://0xacd91124/ -- Google
I did a quick search for whether "fun" Hex strings are reachable, but didn't find any (e.g. the canonical http://0xcafebabe). Random combination of hex-words are login pages to web cameras or cable modems. I didn't try to e.g. replace "e" with "3", e.g. http://0xcaf3babe/
runc - which is the low-component that does the actual container launching in Docker and other runtimes - is mostly written in Go and quite approachable[1], if you're curious what a production-ready container runtime looks like.
Namespaces look simple on the surface, but there are plenty of subleties, particularly when using Go:
- `runtime.LockOSThread()` has to be called before entering a namespace to pin the goroutine to a specific OS thread. The unshare call affects only the current thread[2][3]. Even then, you have to be careful not to spawn any new goroutines[4]. For this reason, parts of runc are currently written in C (you could technically implement it in pure Go, but the maintainers believe it's easier to reason about the C implementation).
- The container runtime has to reexec itself from a copy of itself in a memfd to prevent the container from writing to /proc/self/exe[5][6].
- Various race conditions and symlink attacks during container setup[7][8].
- Some parts of the container initialization have to be done after switching to the new rootfs, which is attacker-controlled territory[9][10].
- ... and plenty of other gotchas, the runc code is full of comments that explain why things have to be done in particular ways.
Obviously, Gocker is an experiment and does none of these things, and you shouldn't run it on anything that you care about :) Sometimes things are complex for a reason.
[1]: https://github.com/opencontainers/runc
[2]: https://golang.org/doc/go1.10#runtime
[3]: https://github.com/golang/go/issues/20676
[4]: https://www.weave.works/blog/linux-namespaces-golang-followu...
[5]: https://github.com/opencontainers/runc/pull/1984
[6]: https://github.com/opencontainers/runc/commit/0a8e4117e7f715...
[7]: https://github.com/opencontainers/runc/issues?q=race+conditi...
[8]: https://github.com/cyphar/filepath-securejoin