How we deploy faster with warm Docker containers
dagster.io
dagster.io
Majority of the time for me a docker compose with my dependencies does the job well enough and is much faster to get up and running with
Dear Valued Customers,
As CEO, I wanted to take a moment to update you on a change to our pricing model. Moving forward, there will be a cost increase for each new function that you write and deploy, as well as an additional fee for each variable declared in that function.
We believe this change will enable us to continue to provide you with the best services while also ensuring our business remains sustainable. Our team will provide a detailed quote for any new functions you request, outlining the costs and estimated timeframes.
We are committed to providing our services to everyone, regardless of their background or circumstances. This change will not affect the quality of the services that we provide, and we remain dedicated to providing you with the best possible experience.
Thank you for your continued support, and please do not hesitate to reach out to us with any questions or concerns.
I've seen the outbound version of this which gets sent to our customers:
Dear customer, extra 20% this year, kthx.Dagster is OSS and most folks do local development on their laptop which is quite fast. The speedups in this post are for production deployments (i.e. landing a commit in your master branch) and for branch deployments (the deployments that run on every pull request).
You can already do local development with dagster, if you set up a local environment. Some users may not want to set up a local environment with all dependencies, secrets and so on, so they can use the remote environment option. Remote environments are also easy to share with other developers and can be used in addition to local environments.
I think the culprit is more the GitHub Actions cache than Docker since it seems to be hard to get a clean cache management. I'm not sure about caching Docker image layers, but caching the Nix store with GitHub Actions is pretty complicated (not even sure it's possible): this means we have to download all required Nix store paths on each run, but i consider this is because of a GitHub Action cache limitation.
So, did you consider using another CI, which offers better caching mechanisms?
With a CI able to preserve the Nix store (Hydra[1] or Hercules[2] for instance), I think nix2container (author here) could also fit almost all of your requirements ("composability", reproducibility, isolation) and maybe provide better performances because it is able to split your application into several layers [2][3].
Note i'm pretty sure a lot of Docker CI also allows to efficiently build Docker images.
[2] https://grahamc.com/blog/nix-and-layered-docker-images
[3] https://github.com/nlewo/nix2container/blob/85670cab354f7df6...
[1] https://news.ycombinator.com/item?id=34898253
Disclaimer: not affiliated, a happy paying customer.
Irrespective of the CI, I believe all cached Docker layers will need to be downloaded onto the build machine before it can be rebuilt.
Still, I believe it is possible to build and deploy faster even with a "docker image only" design and it's something we are still looking at. The question is what is the lower bound here - would be hard to beat "sync a file to a warm container and run it". Pex gives us a pretty good lower bound that is also container platform agnostic.
It depends on the size of your Pex file (i don't think you mentioned it and sorry if i missed the info). With a Docker/OCI image, it would be possible to create a layer with only the Python files of your application (without deps and interpreter). (I say "would be possible" because that's currently not easy to achieve with Nix for instance.)
Docker making some sort of layer-sharing mechanism that constantly distributes layers to all build runners would be worth some cash, I reckon.
At least I’m comparing it to Cloudflare Workers, which deploys fast enough (<1 sec) that I can actually use it as a development environment.
Although they recently announced that Wasmer was being used in their 1.1.1.1 DNS (however, that has little to do with the Workers product I believe)
I agree it would be fantastic to have sub second deploys! Doing this for user specified Python environments is challenging in different ways than doing it for a JS SDK like Workers. Note that just provisioning a GitHub runner takes about 10s, before any deploy code even starts up. In theory we could rsync the code directly to the code server and reload it. But any of these options require bigger architectural changes.
Also you can already have a local environment setup for faster iteration and use this for the shared environments with other developers, where the speedup is still great to have.
Who is using this/asking for this/why?
Building and hashing the dependencies is exactly what adding the requirements file/etc and building a layer does.
> moving around 100s of megabytes of images seems unnecessarily heavy-handed when we just need to synchronize a small change. Consider git – it only ships the diffs, yet it produces whole and consistent repositories.
Just ship the layers, that's literally what docker does, right?
Is this a whole build process just to get around the ephemeral nature of github actions?
Basically, a conventional Docker CI build process looks like the following in what I use (where <tag> is something like branch-foobar):
1. docker login <repo>
2. docker pull <repo>/<image>:<tag> || true
3. docker pull <repo>/<image>:branch-master || true
3. docker build --pull --cache-from <repo>/<image>:<tag> --cache-from <repo>/<image>:<master> -f Dockerfile.<branch> -t <repo>/<image>:<tag> .
4. docker push <repo>/<image>:<tag>
Using --cache-from cuts down dramatically on the build times since (assuming your branch-master tag gets rebuilt nightly) at least the Dockerfile layer steps for downloading the base OS image and installing software can be automatically skipped.
But still, even swapping out the last step in the Docker build makes it necessary to pull the entire image first - assuming your average Java application, that's like 500+ MB for OS+Java JRE+OS dependencies. If you're sure that the change is only in the last COPY/ADD step, you still have to pull the image despite it not being needed technically.
I am fairly sure a docker layer is just a tarfile. If you're just adding, say, python code and performing no execution I feel like you could create the tarfile and add it to the list of layers.
edit - This seemed dismissive of me. It isn't intended like that, I'm genuinely curious as to whether this is the right approach. The post feels intuitively like it's reinventing some things that exist, but perhaps for good reason. I think it'd be nice if everything else was "just docker". Perhaps I'm missing a key point - it feels like just having a build machine with storage would immediately solve the problem, there would be no remote pulling. Is that not a solution?
Indeed. The problem is, creating the tarfile is "easy" but manipulating the image metadata and getting that pushed to the repository is not - you need to implement that all on your own as Docker tooling is useless without the full image.
> Perhaps I'm missing a key point - it feels like just having a build machine with storage would immediately solve the problem, there would be no remote pulling. Is that not a solution?
Hard to impossible to do in ephemeral or shared build services. IIRC, you can't add EBS or EFS volumes to Fargate tasks, and with EBS volumes you don't have concurrent access between multiple build runners.
> Hard to impossible to do in ephemeral or shared build services. IIRC, you can't add EBS or EFS volumes to Fargate tasks, and with EBS volumes you don't have concurrent access between multiple build runners.
You can attach EFS to fargate tasks. My more general point there was that it feels like a lot of complexity disappears if you don't have an ephemeral build service and can just use docker as docker.
Something I'd missed in my first readthrough was about using existing running containers - that makes me a little uneasy but "start from scratch" and "update a running system" are fair points to find tradeoffs between them. It all feels like trying to push the tools to work in a way they weren't really intended to.
There are two sides to the problem: the build and the deploy.
> it feels like just having a build machine with storage would immediately solve the problem, there would be no remote pulling. Is that not a solution?
Indeed, ephemeral builders are an issue and this would be an alternative solution. However doing this for an arbitrary numbers of users who want on-demand deploys for a variety of projects is non trivial. Most CI environments give you a "clean environment" at startup.
Even if the build takes 0 seconds, the other part is re-running this docker image (where 100s of MB is unchanged and say the final 1MB layer changed). You need to ship all these layers to a provisioned container and boot it up. Even this is possible to optimize by heavy caching and optimizing the container service - but that's an alternative solution with different trade-offs. With pex, we can update any docker image in-place so it is container service agnostic. It also works with our current service (Fargate) which has famously slow startup times.
Haven't tested it in a couple years but start time used to correlate with image size (which makes sense). I think the Fargate VMs don't have very many IOPs so you're usually waiting on extracting tarballs for your container to start
With that in mind, it can sometimes be more efficient to ignore Docker later best practices and just try to smash everything down into the smallest single layer possible
Yeah, it was partly that which made me wonder if just removing that would help a lot. Seems weird to keep a clean fresh CI setup and not have that in production
I understand handing off the scaling issues to someone else. Perhaps this is the difference between the "best outcome" and "best solution" as the latter takes into account what already exists.
K8s does some optimisation on image layers but my knowledge there is fuzzy.
Perhaps this is the best solution given the constraints, but it's a shame that the fully supported layering system in docker needs to be recreated essentially due to two services throwing away lots of useful data. A container runner that stops your container, pulls the layers and restarts it would only have the container startup time itself - which is extremely small - and keep everything separated.
K8s may have some more controls for the "incremental deployment" case but I'm less confident about the isolation between pods to run user provided code.
It feels like EC2 is being reconstructed on a platform meant to hide it.
If pods have added privileges or there is a really low risk tolerance, maybe that's not enough isolation.
Sounds like you can change the container runtime with EKS (not sure if that impacts AWS support) so you could use gVisor or runvm
https://www.verygoodsecurity.com/blog/posts/secure-compute-p...
Firecracker is definitely very interesting. Would require more ops work for us to run bare metal EC2 (we currently use Fargate). IIUC reusing pre-existing environments would require us to share ext4 filesystems across the VMs. Not sure if antistrano helps here but will look into it.
Looks like Facebook did it too. I guess there was a phase that tech went through in 2010ish.
[0]https://blog.twitter.com/engineering/en_us/a/2010/murder-fas...
It's much simpler to just point a server to torrent tracker and let them exchange stuff between themselves
using a container instead of a zip for lambda has advantages, but speed is not one of them.
i auto rebuild my go zip and patch aws on every file save.
it’s done before i alt tab, up arrow, and curl.
script: https://github.com/nathants/aws-gocljs/blob/master/bin/dev.s...
Have you considered going full clojure and using blambda as a backend?
jvm vs go is a close call.
reagent vs js is not.
IIRC git does _not_ ship diffs. It copies whole files even for the tiniest change. The compression layer handles the de-duping, which is a different layer.