Hacker News Reader

I'm not sure if Circle supports it yet but you should definitely look into docker multistage builds as that should solve this problem for you.

https://docs.docker.com/engine/userguide/eng-image/multistag...

I've been experimenting with multi-stage builds, and they do let me build a leaner image with clearer syntax by using one the first image to build my gems, and then copying the gems to the final image. <https://github.com/coreinfrastructure/best-practices-badge/b...

However, that doesn't fix the problem of having lots of development and test gems installed in test that I don't want for production. I don't know of a way of copying just my production gems to a production image.

Well, you could load only the gems required for the environment: http://bundler.io/groups.html

Or maybe have more than one path for gems and change load paths between environments?

Or keep tests in a separate image from the production image under test? Similarly, build tools could live in a different image/layer.

It's not a crazy idea to first install production only gems, then copy them (cache them), and then install the remaining gems. But I'm asking for info on best practices.

You can bundle into vendor folders, one for each env and one for the rest.

Do you have a URL demonstrating this, please?

No URL right this second (maybe I should write a blog post–), but I would set GEM_HOME or BUNDLE_PATH env variables as I ran bundler to have it install in a different folder, and add the folders as necessary to GEM_PATH at runtime. I would also consider using a local cache folders for the source of the gems bundler uses. This is the purpose of http://bundler.io/v1.6/bundle_package.html which assumes that gems are installed separately rather than loaded directly by path because of platform differences (packaging on Mac to run on Linux, for example). But you don’t need the overhead of bundler’s packaging and install process if your binaries are built in the same environment they run in, as could be the case with Docker or VMs.

You could also point BUNDLE_PATH at the last place bundler ran for that environment (don’t delete the old gems) and that would speed up the install, but a better approach might be to package to a shared folder and then install to env specific bundle path folders in your docker image/build workspace, if you want bundler to manage the shared caching while keeping gem output separate and unique to each build.

It’s important to remember that gems are just folders of ruby files placed in the load path, sometimes with compiled bits to load at runtime, that could in turn be dynamically compiled to rely on system libraries or non-Ruby packages. So you have similar constraints with Ruby that you would have with Node.js – the potential for issues from the portability or lack thereof for native code dependencies. That’s why there’s a distinction between “install” and “package” – install will make sure the binary output can run on the current system unless flagged not to.

Here’s a node example that demonstrates the same concept.

https://codefresh.io/blog/node_docker_multistage/

You can optionally use a newer version of Docker CE that supports multistage builds.

The "on-the-knee" setup i have been using for my (Django) side project is:

- base image that installs OS Updates + OS package requirements + production dependencies

- dev image inherits from base image, installs dev dependencies + source code volume mounted on '/app/src'

- test image inherits from base image, installs test dependencies + source code volume mounted on '/app/src' (image used both for local tests and CI)

- prod image inherits from base image, adds source code to '/app/src' (this image will eventually be tagged and pushed to a production registry after builds)

Only adding source code to the image in prod image for not breaking cache during development.

If we're honest we have to admit that any difference between dev/test (where you're running tests) and prod (what you're deploying) can introduce subtle bugs either due to configuration errors or something like a missing dependency that you accidentally put in the development group but it was actually needed during production. I've seen this happen and to a large extent it's unavoidable unless you're actually able to run your integration tests with production environment settings (I don't know anyone who really does this.)

On one hand, having dev/test gems in the image should be innocuous, if I recall correctly bundler will not activate them when the environment is production. So other than some additional image size (is this really a problem?) it doesn't have any material effect. But I empathize with wanting to have a completely minimalist prod image, regardless of whether it has measurable benefit.

iron.io has for quite some time advocated having two parallel base images, one dev which has the build tools, and then copying the built production gems to a prod image. [1] If you do this, then the build tools (gcc and the like) are never in the docker image which is used for test or prod. But yes you still need test gems in an image that runs rspec/capybara/etc.

Maybe the real solution then is to take your prod image, and deploy it to some staging environment which then gets treated to a functional test suite. The approach would necessitate running the suite in one container, targeting the application running in another container however. (This is naturally how you would do it if you were using webdriver/selenium-based testing tools.)

[1] https://github.com/iron-io/dockers/tree/master/ruby

Ideally, yes you would want a single Docker image. That's somewhat the point of Docker, to isolate your app and have it run the same, everywhere. If you use multiple images, then you're circumventing that benefit.

You could install all the dev gems, build tools, etc in the CircleCI build container and run unit testing. Once everything looks good and you can build your file app that passes, then take the app and throw it in a new Docker image. Then, integration test that Docker image itself.

That final piece could be done on some sort of staging server of in a new build environment using CircleCI 2.0 Workflows.

I think you might be missing an environment: staging. I would install whatever you want for development. Then once you're happy, push that to CircleCI and have it build the image. Push that image to staging for final testing. Once you're happy with that, push it to production.

I currently push all passing builds on master to master.bestpractices.coreinfrastructure.org, and also run staging and production environments. See https://github.com/coreinfrastructure/best-practices-badge/b...

My question is when switching to a Docker workflow, does it make sense to use the same image for test on CircleCI as for deployment on staging and production.

Typically it's a good idea to separate build and testing phases strictly, exactly for these types of reasons. You deploy the same image to dev, testing and production, and it becomes easy to reason about an artifact without confusing it with the environment it runs in.

I hope this helps.

Right, but the point of my question is that deploying the same image to test, staging and production means that I'm necessarily using a less lean image, since I'm including gems only needed in development and test.

Why do you have gems only needed in those envs?

I'm trying to balance two conflicting goals in Docker: identical images in different stages and the leanest possible production images.

My question is whether development and testing gems should be included in production or not.

That makes sense, thanks. Wim not clear how to break up the gems for different environmenta into different directories.