Hacker News Reader

Comment by c2h5oh | Hacker News Reader

1. Running strip on go binaries is a BAD IDEA: https://github.com/moby/moby/blob/2a95488f7843a773de2b541a47...

Use -ldflags="-s -w" instead

2. Production build should NOT be running glide install - you want ALL your dependencies vendored, locked and commited to your repo before you build it. Bonus: you can have your Docker image build by CI pipeline and know it's going to be exactly like the one you've got locally.

3. If you're including external resources in your container (upx in this case) via url it's common sense to verify GPG signature or, when one isn't available, at least file hash

4. If your app doesn't need things like ca-certs you don't need Alpine - you can just use "FROM SCRATCH" to only have the statically linked binary in your container slashing another 50% off the final size of container.

View on HN

On (2) note that checking in vendored code is not necessary for reproducible builds, as you will get the exact revision that's in your lock file for glide or dep. You will, however, massively bloat your git repo.

I'll take bloat over having Github being up as a build dependency.

Ditto decreased build speed because of internet downloads.

Stick a cache on your build server. As long as your cache doesn't go down you can cache all the build dependencies. Then github is not a dependency anymore.

That does not work with Docker in Docker Builds with the copy from pattern. (Especially not when running on sbt...)

Yes it does. You can pass http_proxy as a build argument to docker and then it will use your cache when it builds docker images.

How far can you take this thinking?

"My app will build even with Github down!"

"My app will build with Github down, and the entire internet down!"

"My app will build with Github down, and the internet down, and the US government being taken over by a foreign military power!"

"My app will build with Github down, the internet down, and the US government being taken over, and Europe a smoldering crater from the last alien invasion!"

At some point - just - nobody cares that your app can build. There's bigger problems.

To me, "Github is down and not coming up!" is one of those "oh my god we now have serious problems who cares about our app" moments.

GitHub usually does a good job but we aren't paying for an SLA guarantee and major outages aren't unheard of. It's easy enough to make your build a deterministic function of the branch contents that I don't see much of a defense for not doing it.

Re: #4

If all you need is ca-certs you can still use SCRATCH and copy copy /etc/ssl/certs/ca-certificates.crt from the build stage image into the final stage

  COPY --from=0 /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/ca-certificates.crt

Thanks, will try this out and update.

> 1. Running strip on go binaries is a BAD IDEA:

This is old information. It is perfectly fine to strip Go binaries and has been for years. This post does a pretty decent job of summing it up.

https://dominik.honnef.co/posts/2016/10/go-and-strip/

On 4, I recommend using the multi stage to build in alpine/Debian/etc and then copy all the necessary artifacts onto a scratch base.

Thanks! Will try, test and update regarding points 1,2,3. My use case required ca certs but I'll mention the scratch alternative too.

#4: Another motivation for using Alpine is to have a shell available so that quick debugging is possible if something goes wrong. For e.g. use ping to see if DNS lookups are happening in a Kubernetes cluster.

I really don't understand the obsession with small docker final images, unless you're doing something silly like export/import and throwing away the benefits of layers. Removing temporary files from layers is great, but you get very little benefit from starting smaller. Especially so if that means your final layer is bigger. That's actually a lot worse storage-wise than having a fat base image and a bunch of tiny final layers.

I do understand the desire to omit unnecessary attack surface, but if you have a good common base image, then you're not incurring any additional storage beyond once per docker host for all the layers your images have in common.

Not a huge docker user but I've noticed various build systems and deployment mechanisms operating slower as a result of large docker images. Faster build times are really attractive to me so I keep my images small.

The major problem with very large containers is that it can cause failed jobs in container schedulers due to the holdoff timer expiring before the container is even downloaded. I've only seen this with Marathon so far but I've not yet tried to launch a 2GB container in k8s.

This doesn't happen on Kubernetes. The Pod stays in ContainerCreating state unless the entire image is pulled. We are running some Java containers which are >2GB and apart from the time it takes to start-up on a fresh node, everything is fine.

When a container is scheduled to a node in Kubernetes, if the image is already available, it takes only ~2s for it to be up and running. But, if it is not present on a node and the image is of 2GB, the download speed is the rate limiting factor, taking start up times well beyond 2 seconds. And in a multi node environment where you do scaling up and down or maybe autoscale, new nodes could come up a lot. So, Leaner images are always better.

> So, Leaner images are always better.

That's a very vague statement, though, given the layer concept. The same final filesystem image can use any number of layers to get there. The way you organize the layers in your images, and share data between multiple apps, is important on top of generally keeping things reasonably small.

What I think you're trying to say is that fast startup is preferable. What you have not addressed is average startup time, which is where you still benefit dramatically by understanding layers.

The smallest average amortized image size is best. That means if you can get all the services for a new node started with 2GB total download, it's better than 3GB total download.

Using layers appropriately, and perhaps adding dependencies used by 80% services to your base image, may be best for overall efficiency. Know the tool and know your usage patterns.

This adds an un-necessary dependency between services. Team working on service A would need to coordinate with team working on service B to keep same layers. The contract should be to keep the image size as small as possible, and teams can work independently.

Also, python:3.6 doesn't always mean the same layers, since it could be rebuilt with entirely different layers with same tag, unless FROM is locked to a particular layer.

I’ve noticed some Docker image registry solutions out there billing / chargeback based upon total image size committed to the repository rather than realized disk usage. This pretty much destroys the space advantages of Docker images completely (I suspect it’s because the underlying asset storage system by such services is super naive or the folks in charge of said companies simply want to charge enterprises boatloads because that will absolutely be noticed by anyone cost-sensitive whatsoever).

We have had our entire kube go down because the image size was huge and the master node only had 20gb disk and docker wasn’t cleaning up old named images. Smaller just extends the problem to the future, but it gives us time to re-build our cluster with a new kube version.

Thank you for mentioning vendoring.

Reply on news.ycombinator.com