jschorr

(Co-founder of Quay here)

Scalability and garbage collection are actually two of the main areas of focus Quay has had since its inception. As you mentioned, most modern Docker registries such as Quay and Harbor will automatically redirect to blob storage for downloading of layers to help with scale; Quay actually goes one step further and (for blobs recently pulled) will skip the database entirely if the information has been cached. Further, being itself a horizontally scalable containerized app, Quay can easily be scaled out to handle thousands of requests per second (which is a very rough estimate of the scale of Quay.io)

On the garbage collection side, Quay has had fully asynchronous background collection of unreferenced image data since one of its early versions. That, plus the ability to label tags with future expiration, means you can (reasonably) control the growth problem around images. Going forward, there are plans to add additional capabilities around image retention to help mitigate further.

In reference to your note: We are always looking for contributors to Project Quay, and we are starting a bug bash with t-shirts as prizes for those who contribute! [1]

[1] https://github.com/quay/quay#quay-bug-bash

Edit: I saw the edit below and realized the bug bash is listed as ending tomorrow; we're extending it another month as we speak!

I've always pronounced it "kway" but so long as its being used, I'm fine with any of the pronunciations :)

Disclaimer: Named, cofounded, and now engineering lead of Quay

Yeah, I figured as much. Just thought it would be fun to add a bit more history, in case some found it interesting. Did not mean to imply that your article was incorrect :)

Some interesting history of source maps: They actually are older than the article suggests, having been original developed and used internally at Google starting in early 2009. At the time, Google had some of the largest applications written in JavaScript running inside of the browser, optimized and minified using Closure Compiler [1]. One focus of Closure Compiler was/is minimizing bytes sent over wire, which resulted in trying to debug issues that occurred on Line 1, Character XXXXX of the minified file. As an engineer working 20% on a Gmail feature, I grew incredibly frustrated with having to add `console.log` or `alert` calls everywhere, and so threw together extensions to both Closure Compiler and Firebug to produce the very first iteration of source maps. While only used by a few members of the Gmail and apps teams at first, the source maps and “LavaBug” (our extension to Firebug to consume source maps) spread internally, so much so that we decided to release them as part of Closure Compiler’s first release (“LavaBug” being released as “Closure Inspector”) [2].

The reason the spec lives in a Google Doc is that it is not a standard; it is a specification that we wrote at Google and saw teams both internally and externally pick up as it gained steam, eventually being supported in all major browsers and most compile-to-JavaScript tools; shows how having a solution to a problem can cause it to grow into a “standard”.

Source: I invented Source Maps

[1] https://github.com/google/closure-compiler [2] https://googlecode.blogspot.com/2009/11/introducing-closure-...

Disclaimer: I'm one of the engineers on the Quay team

Quay doesn't (yet) support the idea of multi-step builds, but it is definitely on our roadmap.

One recommended approach would be to make use of our squashed image support [1]. You can build the artifacts using the in-image compiler toolchain and then delete it (in the same step or a followup step). If you then download the image as a squashed image, all deleted and overwritten files are automatically removed from the layer when it is created. Not a perfect solution, but definitely better than downloading all the unneeded binaries.

[1]: https://docs.quay.io/guides/squashed-images.html

This space is still (fairly) new, so the general answer seems to be that there are multiple solutions to each problem, some that work well with others and some that do not.

For orchestration, offhand the most active projects seem to be Kubernetes [1], Swarm [2], Deis [3] and Mesos[4]. Kubernetes is built primarily by Google, Swarm by Docker and Deis by EngineYard, with each team having experience in different areas (orchestration, containers and full-tier solutions, respectively).

[1] http://kubernetes.io/ [2] https://docs.docker.com/swarm/ [3] https://github.com/deis/deis [4] http://mesos.apache.org/

Kubernetes, Swarm and Mesos handle the orchestration portions only, while Deis is a more feature-complete solution that handles the CI and registry portions as well.

Delivering updates to these solutions and doing so with zero downtime is still very early as well. Kubernetes has a rolling update mechanism, but it can still (occasionally) result in downtime if not setup correctly. Deis handles updates via git-push and will ensure that new containers are in place before the old ones are taken out of service. As for Swarm, my personal knowledge is limited in regards to rolling update, so I'll leave that for someone else to fill in.

For building and delivering images, there are as well multiple solutions. The common solutions are to use a Docker-compatible registry such as Quay [5] (Disclaimer: I'm a lead engineer on the Quay team) or the DockerHub [6]. In addition to supporting simple image pushes, both registries as well support building images in response to GitHub or BitBucket, so they can also be used as an integrated CI, of sorts. Both these services are paid for private repositories. Docker, as well, has an open source registry [7] which can be run on your own hardware or a cloud provider.

Registries are secured by running under HTTPS at all times (unless explicitly overridden in Docker via an env flag), and having user credentials for pushing and (if necessary) pulling images. Registries typically offer organizations and teams support as well, to allow for finer-grained permissions. Finally, some registries (such as Quay) offer robot credentials or named tokens for pulls that occur on production machines as an alternative to using a password.

[5] https://quay.io [6] https://hub.docker.com/ [7] https://github.com/docker/distribution/blob/master/docs/depl...

In terms of how servers know when updates are available, it all depends on which orchestration system is being using. For Kubernetes, we at CoreOS has been experimenting with a small service call krud [8] which reacts to a Quay (or DockerHub) image-push webhook and automatically calls Kubernetes to perform a rolling update. Other orchestration systems have their own means and methods for either pushing or pulling the fact that the image to deploy has changed.

[8] https://github.com/coreos/krud

Hope this information helps! (and if I forgot anything, I apologize)

Also a good list of developers and devops people not getting any sleep tonight :-/

Latest Update from http://status.aws.amazon.com/:

1:52 AM PDT We are actively working on the recovery process, focusing on multiple steps in parallel. While we are in recovery, customers will continue to see elevated error rate and latencies.

It appears EC2 is affected as well now:

12:51 AM PDT We are investigating increased error rates for the EC2 APIs and launch failures for new EC2 instances in the US-EAST-1 Region.

1) Deploy images vs Dockerfiles is always a tradeoff. If you want to be absolutely sure that your code will run in production, then testing locally and pushing the image to a private registry such as Quay.io (disclaimer: I'm a cofounder of Quay.io) is a better approach. If, on the other hand, you want reproducibility of your execution environment, a Dockerfile can be better. As an example: Quay.io itself is built from source using a Dockerfile, which includes as one of its steps a RUN command that executes our full suite of tests. Users can tie Dockerfiles into Quay.io's Github integration [1] so that on every code push the Dockerfile is built (which includes running tests), and then pushed automatically to their repository. If setup correctly, this allows every single code change to be tested and a production-ready image to be created, all without an external CI or build machine.

2) Again, this is a tradeoff. If the context is sufficiently large or changing, it might be better to rsync the data from an external source every time the image is built (or even when it first starts if the data is EXTREMELY large). On the flip side, having the data inside the image means that you do not need to worry about networking and security issues around your data source. One point to note: The cache that Docker uses when building Dockerfiles is sensitive to file changes in the build package; if your files are changing a lot, make sure they are either on an external volume OR placed late inside the Dockerfile to prevent your earlier images from being rebuilt all the time.

3) HAProxy might be a solution for this; you could pipe different requests to different ports based on the incoming DNS name. We are also working on an (WARNING: experimental) project called gantry(d) [2] that makes handling container-based-components a whole lot easier.

[1] http://blog.devtable.com/2014/03/link-your-quayio-repositori... [2] https://github.com/DevTable/gantryd

Offhand, the major benefits for your use case seem to be:

- Speed and Flexibility: Extremely quick turn around time from development to production; Docker images usually start in under 10s and multiple images can be run concurrently. If a user needs to update their running service(s), they can even have multiple versions running at the same time behind something like HAProxy (which itself could be in another Docker container), allowing real-time migration.

- If your users build their images and test them on their own machines, they can then deploy the same images to your production machines with the knowledge that (barring unusual circumstances) they will have the same exact image and dependencies running there. This means no need to write a recipe, less worries about security, and more freedom for your users to use whatever libs they deem necessary (assuming such freedom is allowed).

- If running images directly is not doable for security reasons, your users could use Dockerfiles [1] which still allow you the benefits of Docker but with the ability to review all commands used to create the image. They can even be built in real-time off of GitHub pushes [2] (disclosure: I'm a co-founder at Quay.io)

[1] http://docs.docker.io/en/latest/reference/builder/ [2] http://blog.devtable.com/2014/03/link-your-quayio-repositori...

Here is an article my cofounder wrote giving some background and basic terminology: https://medium.com/devops-programming/7f5fd023158f

In the meantime, if you need to get it working right now, you can build your own binary as outlined here: http://blog.devtable.com/2014/01/using-docker-on-osx-with-pr...

We've confirmed the instructions still work with Docker 0.8 (make sure to change the checkout branch though :))

It is one of the peephole optimizations that Closure Compiler performs to save some space (read: 2-3 characters) in the final output: http://code.google.com/p/closure-compiler/source/browse/src/.... The compiler actually has a lot of cool little tricks like that to save space (since saving even a few bytes for a property like, say, Gmail, can translate to very large numbers very quickly)

However, you can embed extra metadata for your specific source language if you wish, via the extension mechanism. Debug tools can then read the metadata in to do extra neat kinds of mapping (such as collapsing "native" stack frames).