And sorry for gradle2nix, I'm working on an improvement that's less of a hack.
Don't be. Thanks for your work. Excited to learn about the improvement. Can you tell more about what you have in mind?
It looked something like this: https://gist.github.com/takeda/17b6b645ad4758d5aaf472b84447b...
So what I did was:
- link everything with musl
- compile python and disable all packages that I didn't use in my application
- trim boto3/botocore, to remove all stuff I did not use, that sucker on it's own is over 100MB
The thing is what you need to understand is that the packages are primarily targeting the NixOS operating system, where in normal situation you have plenty of disk space, and you rather want all features to be available (because why not?). So you end up with bunch of dependencies, that you don't need. Alpine image for example was designed to be for docker, so the goal with all packages is to disable extra bells and whistles.
This is why your result is bigger.
To build a small image you will need to use override and disable all that unnecessary shit. Look at zulu for example:
https://github.com/NixOS/nixpkgs/blob/master/pkgs/developmen...
you add alsa, fontconfig (probably comes with entire X11), freetype, xorg (oh, nvm fontconfig, it's added explicitly), cups, gtk, cairo and ffmpeg)
Notice how your friend carefully extracts and places only needed files in the container, while you just bundle the entire zulu package with all of its dependencies in your project.
Edit: tadfisher seems to be more familiar with it than me, so I would start with that advice and modify code so it only includes a single jdk. Then things that I mentioned could cut the size of jdk further.
Edit2: noticed another comment from tadfisher about openjdk_headless, so things might be even simpler than I thought.
- it contains information how to actually build the application
- how to set up a dev environment
- how to build application with musl
- how to build application with glibc
- how to build python with only with expat, libffi, openssl, zlib packages
- how to take botocore and patch it up to only have cloudformation, dynamodb, ec2, elbv2, ssm, sso, sts clients
Try to get all of that into a single Dockerfile and see how complicated mess you end up with.
The actual docker configuration is here:
https://gist.github.com/takeda/17b6b645ad4758d5aaf472b84447b...
It might be still confusing to you at first, as you're used to list of incremental steps how to get to the final result, while this description instead is declarative (you're describing not the steps to do, but what the final image should be).
It's basically comparing bash script with bunch of "aws" CLI invocations to a terraform or cloudformation file.
I wouldn't expect a layman to be able to grok that file. That's fine though - it's not for laymen.
> I wouldn't expect a layman to be able to grok that file. That's fine though - it's not for laymen.
This is the kind of comment that makes me want to stay far, far away from Nix and the Nix "community".
I actually pointed out that mkDerivation is something helpful to learn - that's one thing I wish someone made me sit and learn when I first got exposed to Nix. It unlocks a lot.
https://github.com/NixOS/nixpkgs/blob/master/pkgs/developmen...
What that file is doing, is building a package, and it essentially is a combination of what Makefile and what RPM spec file does.
I don't know if you're familiar with those tools, but if you aren't it takes some time to know them enough to understand what is happening. So why would be different here?
I agree with your assertion regarding the language though. I think nix-lang makes it harder to get into Nix.
https://discourse.nixos.org/t/how-to-create-a-docker-image-w...
Wonder if there is a good term for this. I have been jokingly referring to this as 'headed' and headless as 'beheaded'.
I've switched to using a headless OpenJDK build from Nixpkgs as a baseline instead of Zulu, to remove all the unnecessary dependencies on GUI libraries. Then I've used pkgs.jre_minimal to produce a custom minimal JRE with jlink.
The image size now comes out to 161MB, which is slightly larger than the demo_jlink image. This is because it actually includes all the modules required to run the application, resulting in a ~90MB JRE. The jdeps invocation in Dockerfile_jlink fails to detect all the modules, so that JRE is only built with java.base. Building my minimal JRE with only java.base brings the JRE size down to about 50MB, the resulting (broken) container image is 117MB according to Podman.
I've also removed the erroneous copyToRoot from your call to dockerTools.buildImage, which resulted in copying the app into the image a second time while the use of string context in config.Cmd would have already sufficed.
I've also switched to dockerTools.buildLayeredImage, which puts each individual store path into its own image layer, which is great for space scalability due to dependency sharing between multiple container images, but won't have an impact for this single-image experiment.
This is mostly a JRE size optimization challenge. The full list of dependencies and their respective size is as follows:
/nix/store/v27dxnsw0cb7f4l1i3s44knc7y9sw688-zlib-1.3 125.6K
/nix/store/j6n6ky7pidajcc3aaisd5qpni1w1rmya-xgcc-12.3.0-libgcc 139.1K
/nix/store/l0ydz31lwa97zickpsxj2vmprcigh1m4-gcc-12.3.0-libgcc 139.1K
/nix/store/a3n1vq6fxkpk5jv4wmqa1kpd3jzqhml9-libidn2-2.3.4 350.4K
/nix/store/s5ka5vdlp4izan3nfny194yzqw3y4d1z-lcms2-2.15 445.3K
/nix/store/a5l3w6hiprvsz7c46jv938iij41v57k6-libjpeg-turbo-2.1.5.1 1.6M
/nix/store/r9h133c9m8f6jnlsqzwf89zg9w0w78s8-bash-5.2-p15 1.6M
/nix/store/3dfyf6lyg6rvlslvik5116pnjbv57sn0-libunistring-1.1 1.8M
/nix/store/a3zlvnswi1p8cg7i9w4lpnvaankc7dxx-gcc-12.3.0-lib 7.5M
/nix/store/657b81mfpbdz09m4sk4r9i1c86pm0i8f-app-1.0.0 19.0M
/nix/store/1zy01hjzwvvia6h9dq5xar88v77fgh9x-glibc-2.38-44 28.8M
/nix/store/b1fhkmscb0vff63xl8ypp4nsc7sd96np-openjdk-headless-minimal-jre-21+35 91.4M
There's not much else that can be done here. glibc is the next largest dependency at ~30MB. This large size seems to be because Nixpkgs configures glibc to be built with support for many locales and character encodings. I don't know if it would be possible or practical to split these files out into separate derivations or outputs and make them optional that way. If you're using multiple images built by dockerTools.buildLayeredImage, glibc (and everything else) will be shared across all of them anyway (given you're using roughly the same Nixpkgs commit).https://github.com/max-privatevoid/hackernews-docker-challen...
I would be very interested to know where the difference is; is nix including things it doesn't need to? Is the non-nix build not including things it should?
As stated above, I havent done any trimming on the resulting image, so There's too many stuff in the image.