You Don't Need to Rebuild Your Development Docker Image on Every Code Change
vsupalov.com
vsupalov.com
Maybe it's because I'm never confident in the changes I make, but my style is to write a few lines at a time, save, re-run or refresh. I can't imagine having to do a docker build every code change. If you do the later, it's just nominally better than developing on a remote server, which brings its own challenges.
>> I'm never confident in the changes I make, but my style is to write a few lines at a time, save, re-run or refresh.
This is a pretty typical frontend development pattern, especially seeing "refresh."
As a backend dev working in statically typed languages, I will sometimes code for hours without running, and I wouldn't say anything about it is terrible. I haven't worked with Typescript, but it wouldn't surprise me if it enabled a similar development process.
I wouldn't necessarily infrequent running as a technique worth emulating, but in certain situations it works pretty well.
Maybe I'm doing things wrong, but docker volumes are essential in how I like to dev.
The common workaround to this, is to create an empty volume and map whatever you don't want mounted to it. You'll end up with an empty directory in your host system, but its contents are available from within the container.
It has many problems though, mostly permission and ownership related, but it somehow makes Docker for Mac even slower and crash even more often than usual.
Example minimal docker-compose.yaml file for a custom "devcontainer" we use in VSCode that showcases the workaround:
version: '3.8'
services:
project:
image: devcontainer:dev
container_name: devcontainer
build:
context: .
dockerfile: Dockerfile
args:
ALPINE_VERSION: '3.13'
NODE_VERSION: '14.16.1'
ports:
- 3000:3000
volumes:
- ..:/project
- node_modules:/project/node_modules:cached
volumes:
node_modules:
Linux users aren't immune from trouble either. Host volume permissions are a big source of pain--if your mounted files don't have the same UID/GUID as the user baked into the dockerfile then you'll hose access to the files on your local machine. Most simple dockerfiles run as root (especially dockerfiles created by windows or mac developers who never deal with this problem) and all the created files inside them show up as root owned on your localhost. You have to take special care to craft your dockerfile to not run as a root user, launch with explicit user UID/GUID that matches your host machine, and use a tool like fixuid as an entrypoint to fix up mismatches.
A good workaround though is to use docker's internal volumes instead of host mounts. This fixes all the Linux permission issues (the internal volumes live in their own root/docker managed location), and fixes the slowness on other platforms (no VM folder mount slowdown). But you give up the freedom of seeing your files locally and need to go all in with something like VS code remote containers, or running your IDE directly in the container.
If you run this in a screen session it will auto rebuild as you make changes, which is the closest I've been able to come to containerizing my development style: https://gist.github.com/mikedamm/53dc6a78b976eeac88893427424...
Interesting I'll check this script, but I have doubts doing a rebuild is going to be quick enough even with good cache layering in the Dockerfile. Use case is that I have frontend and backend code in one repo, and make changes across them that need to be reflected in the ui.
If the above link caught your attention, you might also enjoy the following ones:
* For the quickest ROI: https://vsupalov.com/improve-your-docker-images/
* Stuff I WISH I knew: https://vsupalov.com/12-docker-facts/
* If your image builds are slow: https://vsupalov.com/5-tips-to-speed-up-docker-build/
Looking forward to join the discussion later!
Maintaining a complex Dockerfile is just more crap that makes programming tedious.
I'd much prefer to just specify in a dockerfile that I need this compiler, these tools and these env-variables, and then I never have to worry about that stuff again.
Basically all of our projects have some level of external dependencies in the form of libraries or binary tools. Some projects overlap dependencies, but on incompatible versions. This gets really hairy.
Trying to do this without containers (from experience, this is what the devs were doing before) turns into a massive documentation project and a nightmare of trying to get all these incompatible things to exist on a single system where we don't even have a consistent target because everyone's local dev environment is different enough to make it a nightmare. And then continually hounding the devs to keep the documentation up-to-date. But even that doesn't solve the problem because "apt-get install nodejs" today is not necessarily the same as "apt-get install nodejs" in six months or a year and nobody's thinking about this so it's just "worked for me last time!". Onboarding people at one point was a _days_ long process and a group effort.
Instead we just re-use the same 30-odd line Dockerfile we use for deploying out to the infra for local dev. Instead of pages upon pages of out-of-date documentation. a bunch of tribal knowledge, and continually running into new and interesting bugs, we now have a short and simple source of truth which _has_ to be up-to-date because otherwise the environment the project is deployed to is broken. Onboarding people takes 5 minutes.
If you're a single person working on a single project, yeah, containerizing stuff is maybe some annoying overhead. Even then I still use it because once you've gotten past the learning curve, it's actually a really effective way to document and define your project's environment and ensure it stays consistent.
1. There isn't nvm for things that aren't node. Some things have an equivalent (e.g., pyenv). Many external dependencies do not.
2. I've inherited a spiderweb of like two dozen different major projects that are all interconnected. Doing an upgrade across everything at once means a stop-work for months while all the developers update tooling, then fix all their issues so everything works, then we try and coordinate testing and deploys of everything without breaking anything.
Or, alternatively:
1. We don't need extra tooling that doesn't exist.
2. We can update projects one at a time instead of mass-migrating several million lines of code across half a dozen different frameworks.
Now you have bunch of new and interesting problems with docker ecosystem.
Newer tools are much much better in this cross-dependency problem as they evolved too, not only docker. I don't remember dependency hell nowdays while it was more or less common 15 years ago.
I had the same project with and without docker and I totally prefer without docker. There are few quirks here and there but problems are mostly solvable within minutes unlike basically anything you can get with docker.
In practice you'll still use Docker to deploy because it's a nearly-unavoidable deployment protocol at this point. But I see very little reason to have it house your in-progress code in these cases.
1. Configure a service 2. Use start command sleep infinity 3. Install inotify_wait on windows 4. Do a loop like this
Look for changes, Rsync changes to pod, Pkill java, Run start.sh
We do a grep on changed files and only kill java if jar/classes and other deps has changed, that makes it possible to edit html in pod and get fast updates.
It’s essentially a vastly superior “vagrant up” on anabolic steroids
We recently re-wrote the hot reload functionality to use Mutagen[2] under the hood and it's insanely fast (<200ms anecdotally). It also does two way sync which can be useful. The old implementation used rsync but a lot of our Windows users struggled with that. So I figured I'd share in case that sounds familiar.
What happens after a sync event depends on the stack, but we've had pretty good success with Entr[3]. We often have it watch a single file so that multiple watchers in a shared dev cluster don't eat up all the node's resources.
1: https://github.com/garden-io/garden 2: https://mutagen.io/ 3: http://eradman.com/entrproject/
There are caching settings to make it better, but there are race conditions between the watch and when file contents change, so sometimes the JS stack will compile an inconsistent file.
OTOH, on linux, it's grand.
In the past, to make a very similar workflow possible, I've built tools that automatically watch your source files and rebuild & restart only what is needed [0]. This was built for bazel + docker-compose but there isn't a reason one couldn't watch the "build:" contexts for what files are important.
At a previous company one of our engineers was a huge fan of this volume mount approach and every single time something broke (which was very frequent due to some prod/dev env magic we had) I had to assist quite a few more junior devs figure out what was wrong with their machine. For those with scripting languages, was it their system's newline endings? For compiled languages, was their system SDK different then what was in the container? For prod bugs, did they forget to rebuild & test the container before opening their PR (we had no automated integration testing)?
In my opinion, if you can make your build system in charge of building/packaging things you'll have a much happier time.
If you're using a linux image that expects a directory like `UpperCase` and you named it `uppercase` locally it would work.. until you bake the source for your production release and you get errors around that directory not existing.
"But it works locally!"
0: https://www.jetbrains.com/help/pycharm/using-docker-compose-...
Can be used to build scratch containers. Can be used to cross compile.
The only major downside is it requires learning and understanding nix which I get is a hurdle, but one that's well worth it.
Docker is only one container tool of many now, and it's worth exploring what else is out there.
volumes:
- .build/.bundle-cache-dir/app:/usr/local/bundle/Same as in frontend for example you don't "npm build" your way through development, instead you want hot-reload and other similar features.