Hacker News Reader

I'm working at a shop where we have to use extremely large images (10 - 40 GiB) on a platform of 60+ medium sized blades (4 - 12 cores, 32 - 256 GiB RAM, 200 - 1000 GiB SSDs). Startup time for running such huge images often is 5 - 10 minutes. They contain prepackaged test data so that they can be run in parallel against newly built versions of our software - there are about 30ish of these images.

You not only store these images once in the registry, but on all container hosts that they might run on. And you need to transmit them, if they aren't cached there, yet. And you may want to redeploy new images several times a day, as they get updated, new ones added, etc.

To be fair, the above is not the use case containers were originally intended for. And if your 10 GiB container image only runs on 3 or 5 nodes and only gets updated once a week, you obviously don't have to worry about the overhead too much.

But at scale (beyond 10 nodes) and/or on very fast development cycles (=more then one deployment per day), size IMHO starts to matter.

I can totally see this starting to matter, but I do want to point out that it hasn't ground your business to a halt just yet and is very very unlikely to. From what I've seen most ops pain comes from a lack of automation when deploying -- as in if the process is 5-10 or even 15 minutes, no one really cares (outside of an emergency) if it's completely automated. Do you find that is true with your work as well?

I can certainly see that it can be a painful problem (10-40GiB is huge), but I expect most smaller shops that aren't running on their own servers/colo as such these image sizes were never even an option (I can't imagine trying to upload 40GB across the public internet to launch one instance!).

I've worked in the general automation space for awhile, and agree that time-to-result is rarely the limiting factor.

It's also not usually total-person-clock-time.

It's number-of-distinct-manual-steps.

(5 minutes manual) + (15 minutes automation) + (5 minutes manual) + (30 minutes automation) = everyone hates doing it

(10 minutes manual) + (45 minutes automation) = everyone's happy and can use their time productively

Because every manual step means things can go wrong. You might also need to look it up if you don't do it often. All these increase the overhead of manual steps compared to clock time.

Partly. I think it's also because manual steps require synchronization.

Not only do I have to do something, I also have to set an alarm / watch the clock to know when to do something.

And the CI with many-GB Docker images is very painful... Turning on layer caching usually makes the process even slower as it needs to pull and unpack the previous image before starting, and if you turn it off you're downloading a ton of deps on every build.

If you separate the heavy stuff into a base-image you still have to load it on the beginning of CI, which without beefy machines with local SSD caching can take a loooong time.

So, is there a point where it makes sense to use virtual machines instead of containers?

For this particular use case: We previously used VMs and snapshots for that workload. The problems we encountered were: - snapshots aren't really intended to be portable (we used ESXi and also KVM on LVM backed volumes), hence we had to write and maintain tooling to have a "snapshot repository", versioning for these and distributing them to the target nodes - that did all work, but was even slower (startup times, which can include shipping and setting up the snapshot, was 15 - 60 minutes) - a VM will duplicate all of the services and the kernel, so all of that has to be started as well, where as the containers only start the services under test. - Using VMs makes it much more difficult for our developers and QA to retreive a particular image and replicate a failing test on a given version of the tested software locally, especially when considering a wide range of client OSs we see in our not that large group (MacOS, various Linux distros, even Windows)

A general observation on trade-offs: With containers layering and COW you get fast development, but have to pay the performance bill when you happen to download and apply an image for the first time. Similarly, taking a snapshot of a LV under LVM or on a ESXi VM is fast, but applying the snapshot is slow.

We had therefore at one point considered using Ceph RDB and their COW clone snapshots[0]. It would let us do "cheap" restores of snapshots. Our initial tests showed that the network bandwidth requirements[1] would have needed some serious infrastructure re-engineering in order to keep up with our I/O expectations. And again, the containers slot in nicely with commonly available local resources and allow working offline, to a degree.

[0] https://docs.ceph.com/en/latest/rbd/rbd-snapshot/ [1] the environment in question uses 10 - 40 Gb/s network links, 40 on the gitlab git and container registry side, 10 on the blade side.

I appreciate the response and you answered my next question about the networking specs!

What if you had a way to use Docker to create your VM images? Then you'd have no registry, be able to mount the vm storage over network(ondemand fetching), boot quickly.

I don't know about you but I work on a laptop, and I work on multiple projects, and each project might use dozens of containers for builds, testing, releases, etcetra. If every container were a gigabyte in size, majority of the laptop's internal storage would be wasted on them, nevermind the time wasted pushing and pulling updates.

I do too (my laptop is pretty beefy though), and yes you're right this would be a problem, if containers weren't built to combat this with Copy on Write filesystems.

That said, dependencies do take up a lot of space, but machines these days also come with a lot of space. People also run prune and develop caching strategies (ex. NPM) as a result. It's not like every container will be a GB in size, as it's not reasonable to have every container have that much content, but how many people are out there using 950MB dependency-filled base containers with 50MB of actual relevant application code? Most people learned the run-it-on-alpine lesson like 5+ years ago at this point.

My new laptop has 3T of disk. I have the old feeling of what can I possibly do with it all. I have two copies of all my photos and text files etc. and dual boot Arch and Ubuntu but still it is so Empty. I don’t think even a very aggressive go get campaign can fill it.

My work laptop has 0.5T SSD and it's 90% full. Turns out VMs, large repos, and OS projects are big. I know there are some large docker images too.

Do you run `docker system prune`/`docker image prune` and related commands from time to time? I assume you already know about it but just in case.

Just recently I found that I had like 100s of GBs used up on my hard drive because I'd installed and used podman for a while and it's image store was separate and just... sitting there, long after I'd moved back to docker for reasons.

Why create bloat when it can be avoided? The 6kB server might be overkill but the other options in that article seem pretty good.

A lot of companies that sell products (non open source ones anyway) actually try to bloat the binaries and deployments to make them look more significant. Many gigabyte installers are not that weird for some products, where I found (for fun as corp wise we cannot really care) the relevant software we are paying for to be a few mb and it doesn't need or even touch the rest the rest in the package for 99% of cases. But it seems/feels like you are paying those $100000s for some serious big software.

When software was still delivered in physical boxes, I noticed the same thing and I even found products in the past where the .exe had megabytes of random data attached to it to make it look bigger.

With Docker images it's all a lot easier to bloat things up and 'hide things'; I used to pack up our main product in a chroot tgz (this is early 2000s) with an installer, so I did not have to actually think about different linux distro's or installers; I know for a fact that some people bought our product over competitors because the installer was so large in size so it looks like better value. People told me when I asked them afterwards why they did not go for the competition. And this was not intentional; it just had a full Debian install in a chroot.

This was something I never considered, but reminds me of the Larry Ellison tactic of never naming anything v1, or the concept that because something costs the most, it therefore must be the best.

Well, Oracle was (not sure if they still are) one of these companies. Some of their products we used in the past where artificially bloated and packed over multi CDs etc to make it all look bigger.

Oracle was cofounded by... Larry Ellison.

If all the readers don’t just know this, I am old and full of useless knowledge. A good read to understand Ellison’s persona in the 90s is here: https://philip.greenspun.com/wtr/dead-trees/53011.htm

Particularly flinging $100 bills at peasants and how RDBMS set pricing for their products. If you do have an implicit awareness of Oracle and their tactics, you will find this an amusing piece of nostalgia when people were starting to understand just how to build efficient and nice web sites in the 1997 time frame.

Speaking of 90s, remember the Greenspun AOLserver courses?

I loved Greenspun. So funny but so clear on how to make stuff work efficiently. My first threaded program I wrote professionally was a n extension to AOLserver to let the TCL layer speak the AOL flap protocol back to the backend. Got it working in few days but couldn’t get three simultaneous threads working for like a month. My boss at the time was like (once I finished it) don’t worry, software is late. My first project in the no threads server framework took six hours start to finish.

> why are people obsessed with container size

Good question! Security should be a priority instead.

A lot of containers on docker hub have serious vulnerabilities, as proven by various research papers. Removing bloat is good for security.

However, using a well-tested distro and receiving security updates from its security team is more important than having a tiny footprint.

This is a good reason to use Debian stable in containers and avoid alpine.

At scale every tiny bit helps. Especially now that Edge computing is becoming a reality, it's much better to pull and run a 1MB container image than a 100MB one on limited hardware and bandwidth

Agreed -- edge computing is a different world, and the difference between 100kb-1mb-10mb-100mb is huge. What I was discussing is definitely not a fit at that scale/usecase -- I'd argue that most infrastructure using containers (and mos t people who see this blog post) are not built for this scale/use-case, and I'm not sure containers even make sense in the edge computing paradigm just yet.

Plus, figuring out optimal sizing for Docker images is in general complicated and often counterintuitive. A company I worked with mandated that every image must be written from scratch to reduce size on disk and save on transfer bandwidth/time. The end result though was that there were no sharable layers, and so every server had to redownload every image wherever there was the tiniest update. Moving to a set of common 1GB+ base images with all dependencies included made the entire system smaller and faster.

Wasted energy and bandwith do matter. Now, there is a gap between a few kB and multiple GB.

oh god I fell in love with everthing related to Alpine Linux. Found out about it when I started playing around with docker but then ported all my home servers and Raspberry Pis to Alpine. It's so damn small and has little overhead. Need docker? `apk add docker`, done!

I've written guides on how to boot Alpine (even with GUI) over PXE [1] and how to set up a perfect file server that runs from a RAMdisk USB Thumb drive with full disk encryption [2]

[1] https://blog.haschek.at/2019/build-your-own-datacenter-with-... [2] https://blog.haschek.at/2020/the-perfect-file-server.html

https://rom-o-matic.eu/ is a broken link.

https://blog.haschek.at/2020/the-perfect-file-server.html is made obsolete by SmartOS, which does the exact same thing, in addition to offering Triton (enterprise web GUI for virtual systems), OpenZFS (end-to-end data protection, ease of administration), zones (full-blown, running-at-the-speed-of-metal virtual UNIX servers), DTrace (for production-safe, real-time, deep machine state instrumentation and inspection), Bardiche (for building firewalls), FireEngine / Crossbow (high-performance TCP/IP stack for building virtual switches and routers) and imgadm(1M) / vmadm(1M) (for virtual server provisioning and software management), and last but not least, a reference implementation of the NFS V4 protocol.

> is made obsolete by SmartOS

or by proxmox or by any other OS. That's not what it's about. If it comes with a web gui, it's already bloated for me. This project was about the minimal perfect setup for my needs and I only need SSH access

SmartOS does not come with a web GUI; Triton is completely optional and isn't required at all, but if one is running 100,000 servers or more, it's there as a gratis option if one needs one.

As far as I am aware, and please correct me if I am wrong, but "proxmox" does not have nearly the same list of capabilities as SmartOS, all the while running from a ~650 MB, read-only RAMDisk.

You seem to have jumped the gun a bit. The question was "why are people obsessed with container size?" while you're answering some question like "What's a good and small container OS?"

Yes kind of but also not really -- this answer was really delightful to read because Alpine linux literally was the first answer to the first concerns with container size. Before docker multi stage builds existed (people we doing the builder pattern adhoc before that anyway), the usual way to get drastic size reductions in container size was to run your container on alpine linux.

Base Ubuntu/Fedora used to be absolutely huge, and no one was smart/patient enough to pick out all the dynamic libs that you needed with your JAR or script to go into production so you usually just shipped the whole fat image.

Yes yes, but you also seem to be missing the point. Here you are answering the question "How did Alpine appear and what came before it?" but again missing the original question which was "why are people obsessed with container size?"

Alpine is one of the unsung heroes of the container world. It's insane how much value Natanael Copa has created & shepherded over the years (decades?). Recently came upon an interview with him and it was the first time I saw the creator behind Alpine Linux[0]. Similarly, the musl libc project, just chugging along, giving most projects that build on top of it an easy out for portable static binaries.

Thank you for those links, I am about to gobble up those blog posts -- I recently went on a benchmarking kick[1] and diskless alpine instantly struck me as the perfect server setup. ECC memory + running from ram would give me full use (to put in RAID/whatever else) of the NVMe drives, it's something I'm going to try out as soon as I get a chance to.

I am so interested in the infrastructure space, I know exactly two hosting providers that will give me PXE level access (so I could use something like tinkerbell[2]):

- OVH [3][4]

- Vultr[5]

- LeaseWeb[6]

Unfortunately my personal favorite hosting provider, Hetzner[7] (I fell in love the moment I came across the robot marketplace) does not offer it yet, though I've automated going through their rescue system at this point so it's OK.

[0]: https://www.tfir.io/meet-the-creator-of-alpine-linux-natanae...

[1]: https://vadosware.io/post/k8s-storage-provider-benchmarks-ro...

[2]: https://docs.tinkerbell.org/

[3]: https://github.com/gmasse/ovh-ipxe-customer-script

[4]: https://geekgonecrazy.com/2020/09/07/tinkerbell-or-ipxe-boot...

[5]: https://www.vultr.com/docs/ipxe-boot-feature

[6]: https://kb.leaseweb.com/products/dedicated-server/installing...

[7]: https://www.hetzner.com

It's too bad that in the path to minimal disk size they decided to simply disregard legal obligations and they ended up producing something that violates the open source licenses of the contents... so unusable for any product.

I am working in Manufacturing Execution Systems area and the internal sites are spread around the globe on all continents; speed does matter, there are dozens of places hundreds of kilometers away from a place with Gigabit Internet (for WAN). Taking backups off site is faster by physically shipping hard drives than copying over the WAN, I don't like any large software.

In one place in Africa driving to another site 3 hours away gets network speed 10x better, so we do this from time to time.

Container size generally impacts startup/deploy times. 500MB container still takes almost a second to copy over gigabit connection, and typically you need to copy it at least twice: from build server to repository and from repository to whereever its going to run.

A Gigabit Ethernet connection means 1Gbps minus some overhead, effective rate is around 117-120 MB/sec, so the 500 MB container takes at least 4 seconds to copy. 10 Gbps gets you to less than a second, but that works usually between servers, not from the regular laptops.

Which also means that _if_ you pull at full speed, this node has no bandwith at all for serving users (more or less).

I wonder the same thing. I can see the value in shrinking a 1GB container by 90%, but I suspect there are diminishing returns by the time you're shrinking a 5MB container below 1MB. In most organisations, there are probably bigger opportunities for improving security, performance, etc.

It's worth remembering the MBs add up in multiple places. Sure, it may not be worth for 5MB. But once you're in tens/hundreds, they add up in storage (how many old versions do you keep), transfer times (how quickly can a fresh instance pull all tasks and how quickly can a developer (at home rather than in us-east-1) start with a project), transfer costs (how many times does a CI node need to download/upload the last layer).

Saving 10% in size may actually translate to a non-trivial cost saving - both obvious and cost-as-in-paid-waiting-time.

Also all those dependencies are security risks. A lot quicker and easier to scan a container with just one go executable than even the most benign distribution of core utils and so on. And if your containers are just one binary and you can read the logs then you don’t need to go into the container any more than you need to go into your binary. Then your k8s node or other container runner host is just like your old host running a bunch of processes but with much better process isolation and file system isolation.

There are some teams where this will be the best use of their time, and I'd be well out of my depth suggesting to them otherwise.

But the vast majority of teams in my experience have spent ages farting around with these kind of micro-optmisations, whilst there were far better opportunities to improve the infrastructure. And in the process, they actually made it more fragile and harder to debug.

Simply put, this is a niche skill with niche value. Most people trying to do this probably can't and shouldn't do it.

> building the code closer to the deployment area is also a solution

This is exactly what we do. Once the first build of our application is installed on our customer's servers, it rebuilds itself from source each time. This is also really good at firewalling CI/CD compromises away from your customers, as long as you don't merge cryptolockers/miners into your private github repos.

This.

I found that most people focus on size and forget about reuse.

Sure, each layer adds size to your final image, but you can reuse them, so that you don't need to transfer the complete image but just the layers on top of the one that changed.

Flatten everything and you need to ship the whole stuff.