Speeding up LXC container pull by up to 3x | Hacker News Reader

Speeding up LXC container pull by up to 3x

databricks.com

97 pointsctecte22 comments

A long time ago, I wrote something similar ( https://github.com/htcat/htcat) to assist with Heroku's efforts to speed up moving the tar formatted application releases around. It's pretty old, it doesn't integrate tar archival itself, it probably can stand improvement. Or given its small size, even a rewrite.

My favorite hack in there (that also made it work with pre-signed S3 urls) was not using the HEAD method as is customary to determine object sizes, but instead doing a regular "GET" that, for small files, would execute on its own...but for larger would simply be abruptly closed by htcat once it reached the bytes that had since been fetched in parallel by a range-based request sent immediately afterwards. The goal was to have htcat not offer a penalty on small files so it could be used on blended workloads without thinking.

It also found a bug in S3's range implementation. We had a problem with some object or other, I wrote in about it, and was told that upon investigation a bug had been fixed. No more problem.

Some kind soul packaged it for Debian, so it's easy to get if you need it.

Oh nice! Yeah this is basically the same design/intuition that this project also started out with (I also just wanted some way to combine the benefits of aria2c with piping). Wish I found this earlier lol.

I only later decided to look into also parallelizing the tar extraction as well. This originally started with looking into making a change to the GNU tar source code itself. However upon cloning I realized there were a lot of design choices in tar that made the assumption of single threaded execution (global state, etc). At this point I decided it would just be easier to re-implement the tar extraction myself using golang's "archive/tar" package.

How about squashfs files for every layer and overlay mount those. At the end of the day you access probably .1% of the data, so with this you'd defer decompression to runtime and only decompress what you need

Had not considered that. Reading about squashfs I see that it's readonly. Do you know if that's fine when mounting the directory for an LXC container? Also I wonder how much performance impact (if any) there might be on spark jobs due to the decompression at runtime.

To be completely honest this all started as a quick hackathon project just to speed up downloading before piping to GNU tar. Only later did I consider also re-implementing the tar extraction in a parallel manner. If considering changing the overall packaging method from tar to something else there's a lot more ways to consider going about this (including squashfs).

I'd say one of the nice things about this is that tar is fairly ubiquitous, not just in container images. For example, we also have tarball build artifacts in Jenkins jobs that could benefit from this tool.

You can cut down the startup time to milliseconds

1) either mount tar directly

2) or use squashfs and mount that...both would be mounted with some sort of rw overlay

3) underneath do some lazy httpfs type filesystem, so filesystem can be mounted while it's downloading

4) parallelize the underlying download ala aria2c

5) provide metadata to aria2c-alike downloader with file boundaries, so extract further latency/bandwidth savings

1) Are there any ways to do this in a performant manner? Afaik, since the tarball is still sequential with no way to jump around, interacting with this filesystem would be fairly slow right?

2) Agreed, still need to look more into this, although it's more involved and entails changing more of our pipeline for packaging and distributing these images.

3) I need to look into how this would handle files that haven't been downloaded yet. From what I know about httpfs filesystems, I'm not sure how much would need to be done to let this block until file needed is downloaded vs the normal behavior of calling out to get the file being requested.

5) Could you go into more detail here? Not sure I understand how file boundaries, etc can help with latency/bandwidth.

Ad 3. With squashfs via httpfs it would probably be fast enough. I remember years ago httpfs capable of booting livecd iso and it was performant enough. I'm not current with this knowledge, but is there a httpfs, that would download increasingly cache the file while doing range requests as it gets read requests for certain parts of the file?

3) it just blocks and prioritizes that part of download 5) if you can chuck the file so chunks that get downloaded are whole files..less likely to do multiple requests for a single file

6)I forgot to do the best part of the optimization..do feedback-guided-optimization. You can then repack the squashfs/tar file to have all the frequently-accessed files together

Ahh got it, this makes a lot of sense, maybe something to try next hackathon :)

Something to consider, if you are IO constrained, compression may speed up reads because you shift some of the cost of IO to the CPU.

Ultimately, you'll need to measure this to know for sure, and those results will likely only be valid on a given hardware configuration.

OverlayFS also has a "copy_up" function, where the file is copied at the initial write. Once the copy is done, I'd expect write access to be fast. Again, you'll need to measure this.

The setup could probably look like:

container read/write -> OverlayFS([mutable fs as overlay] -> [squashfs layer as underlay] -> [squashfs layer as underlay])

Came here to say exactly that, squashfs with overlay mount for writes beats this easily.

Hard to change the entire container ecosystem away from tarballs tho so this is still useful. Is it available anywhere? I didn’t see a link?

Company wanted to see if there was any interest in it before putting in effort to open source :/

Hoping this is enough to release it on Github.

Have thought about writing the same thing, would love to see it OSS.

Real value for me would be to integrate it with containerd or similar to speed up image layer pull times, I imagine that isn’t shelling out to a separate binary tho.

What language is it implemented in?

This was written in golang. Specifically channels and io.Pipe objects made it pretty straight forward to "glue" different phases of download and extraction together into the final solution.

good fit for containerd then =)

This is interesting and seems general purpose. Not merely for container images.

There’s this option for OCI containers which I don’t pretend to understand: https://github.com/containerd/stargz-snapshotter

It is used by containerd and nerdctl. You do have to build the image with it. Images work in OCI compatible registry. By fetching most used files first container can be started before loading is finished. Or so I gather.

That's pretty interesting. One downside might be losing the ability to do checksums on what's been downloaded, as a safeguard. I wonder if there's a way to preserve that without ruining the parallel downloads and un-tarring.

I did look into performing stuff like md5 inline while streaming the download, however it tended to slow down the extraction significantly. Other less rigorous checks like crc32 are still possible inline though with minimal performance reduction.

It should also be possible to split off and write the raw download to disk in the background with something like an io.TeeReader object [0]. You could then checksum it after the fact like you normally would.

[0]: https://pkg.go.dev/io#TeeReader

GNU tar seems to auto-detect compression type as well. I don't bother passing a flag to say which type to use, when I'm extracting.

Author here, it's true that GNU tar auto-detects compression when specifying a local tarball to extract. However when piping a file stream to tar to extract it actually doesn't auto-detect and fails if you don't explicitly specify a compression type. Should have made that more clear in the post.

Oh, that makes sense. Nice addition.

Tangentially related - do any shops still use LXC containers instead of Docker/K8s?

Reply on news.ycombinator.com