Hacker News Reader

Comment by paulfurtado | Hacker News Reader

FWIW, if you wanted to get the static from-scratch go image a lot smaller, try running the binary through UPX. It typically improves compiled go binary sizes by absurd amounts: https://upx.github.io/

Also, https://busybox.net is a 1.0MB static binary that provides all of the standard posix-compliant tools (sh, grep, tail, etc) and actually comes with an httpd implementation. So if you wanted a more useful container at about 1MB, you could use it.

Edit: here's a previous thread on UPX that has 80 comments talking about the tradeoffs it makes. You should definitely consider these if you intend to use UPX: https://news.ycombinator.com/item?id=15456980

View on HN

Keep in mind that UPX needs to decompress the whole executable to memory and thus cannot make use of OS-level memory sharing. Normally, when you load the same executable twice, its base image will only consume memory once. But it with UPX, you cannot make use of that. So in the end, it may use more resources than you thought!

Definitely important to note, I wonder if upx could solve this via execing an memfd_create in some mode. Also wish upx came with a trivial extraction tool so you could use it for distribution but not runtime.

Aren’t your container layers compressed in transit anyway?

UPX does significantly more compression on binaries than gzip can on the layers. It's not just simple compression, it makes a lot of modifications to the binary to make it more compressible.

Perhaps this is what you were asking for in the GP, but it would be interesting then if UPX could apply those transformations without actually doing the final compression step itself. Perhaps the win isn't as great without the ability to smuggle specific hints to the compressor, but it would be interesting to see.

On the other hand, if it could be demonstrated that modest increases to binary size correlated to significantly improved compressability, then maybe that's something that should just be considered by the compilers themselves.

I was initially a bit skeptical if the page cache would work with all layering docker does. But OverlayFS, perhaps unsurprisingly, does indeed support that: [1] "OverlayFS supports page cache sharing. Multiple containers accessing the same file share a single page cache entry for that file. This makes the overlay and overlay2 drivers efficient with memory and a good option for high-density use cases such as PaaS." TIL

[1] https://docs.docker.com/storage/storagedriver/overlayfs-driv...

UPX creates anonymous pages that contain the decompressed payload, right? Those aren’t file-backed pages so how would the filesystem even know those pages exist?

That’s correct and I guess only page deduplication might help in that case. What I meant in my post was just to state that docker can use the page cache despite the massive indirection caused by overlays. I was wondering if that was even possible and it indeed is.

that's true, plus on low-end devices upx will take forever to decompress and is essentially unusable.

Is UPX actually that much slower than extracting a gzipped docker layer? I didn't realize UPX decompression was much slower than other compression algorithms

The base would consume “up to once”. You will only load pages touched, and they are all reclaimable under memory pressure. Memory used by UPX for decompression is guaranteed to be used, and is not reclaimable!

So in fact, it’s not might use more resources, it’s guaranteed to use more resources.

The only benefit of UPX is saving disk space, which is unlikely to be super useful to most people these days.

The other use case is to distribute images more quickly over the network and improve image pull times, at the expense of slightly more memory use.

Similar to how saving disk is less useful to most people these days, RAM on servers in big docker/kubernetes clusters is also likely not much of a concern for binaries that are well under 100MB if you're not running a ton of copies on each server. In particular, it seems like this tradeoff is most useful for static go binaries, which don't benefit from shared system libraries and aren't likely to have many copies per server, but of course that definitely depends on the use case and I definitely do not recommend blindly using UPX without considering the tradeoffs.

It would be an amazing integration though if it were possible to integrate UPX with docker's layer compression such that the binary is extracted to the FS before running. But given how it all works, that's pretty challenging to implement and the network is only getting faster...

Are you sure UPX actually beats the standard layer compression which is already provided?

After applying UPX, the result should be nearly uncompressable, so if UPX performs worse than layer compression then it might be better for network usage to not use it at all.

In this scenario where you are unpacking it in advance, that eliminates the main benefit of UPX which is that it is self-extracting.

Beware that at least historically some antivirus software would wrongly classify any UPX-packed binary as malware. Don’t know if this is still the case but something to keep in mind.

"some antivirus software would wrongly classify X as malware" is true for any X (except malware, then would not be wrong)

I did exactly this a few years back.

It added 18.5 kB on top of busybox, so, not as good as the 6kB ASM server code. With busybox included, the whole image is 1.18 MiB, or 751 kB compressed.

If anyone cares to take a look: https://hub.docker.com/r/swalog/tiny-hello/

Hi, author of the article here!

I figured there were a bunch of ways to shrink that one, and I had a C based image that was in between my GO and ASM images, but decided to cut them for the sake of brevity!

stealing vexu's routez: https://github.com/Vexu/routez

gives you a multiprocess http server in 612K uncompressed if you compile -Drelease-small

About busybox: if you want a small container, you don't need the posix tools at all, but just your binary. No shell.

Be sure to check out the switches, --ultra-brute is what I usually use.

If you only want the sh, you can get it even smaller. This doesn't UPX the busybox bin, though:

$ ls -shL `nix-build -E '(import <nixpkgs> {}).pkgsCross.musl64.busybox.override { enableStatic = true; enableMinimal = true; }'`/bin/sh

128K /nix/store/qkq9cj81mcvnfr5y04jmf6nqvnrfg2z1-busybox-1.31.1-x86_64-unknown-linux-musl/bin/sh

1.0MB is quite large compared to 6.34kB :)

Certainly, I think this is more about getting over the developer pain threshold. In terms of computer speed vs human perception I don't see either one mattering too much (compared to a 900meg base for node-debian on Docker, although I guess there is some incremental stuff for that?).

It definitely is, although a cool thing about busybox is that you can rebuild it with any applets disabled. I'm not sure just how small it can get though.

But yeah, the tradeoff of that 1MB is that you get a very functional shell environment vs just an http server. It's impressive just how small ut is, given everything that it is capable of

> It definitely is, although a cool thing about busybox is that you can rebuild it with any applets disabled. I'm not sure just how small it can get though.

They do distribute statically linked individual binaries [0]. For just HTTPD and nothing else is about 85kb. But it is a lot more than just a static file server - it comes with CGI support.

[0] https://www.busybox.net/downloads/binaries/1.30.0-i686/

> But yeah, the tradeoff of that 1MB is that you get a very functional shell environment vs just an http server. It's impressive just how small ut is, given everything that it is capable of

Maybe.. I suspect a well-designed Forth image could pack similar capability in less than 100kB. So it might seem low, but not a miracle.

If you're going for size, why not just sidecar all of the "standard posix-compliant tools" but ones that you would want normally. For example, bash instead of ash/dash. You don't need those tools all the time right?

Well, you don't even need a shell.

I mostly meant to sidecar the shell and treat it as a debugging utility. While I get the point of ash being smaller as a point for busybox, if size stops being a concern because the shell is in an inactive-sidecar, then you can use a shell that has less surprising edge cases for users, like bash.

Reply on news.ycombinator.com