LXC – Running 14,000 tests per day and beyond
codeascraft.com
codeascraft.com
It looks like they're not leveraging LXC via Docker. I wonder if that's because they've been doing it this way pre-Docker, or if there are some technical reasons why it made sense to skip it.
Docker is great when it fits your use-case, but there are lots of practical ways in which LXC stands great all on it's own.
I've commented on HN before about why I often choose plain LXC over Docker: https://news.ycombinator.com/item?id=6378823
And also the lxc-user mailing list is a great resource: https://lists.sourceforge.net/lists/listinfo/lxc-users
It becomes more interesting when a) your system has been running long enough that you've learned operational and design lessons that justify an upgrade, or b) you're starting fresh on a new architecture.
Most of the Docker community is made of these groups A and B. If you're starting from scratch, or if making significant changes anyway, it makes a lot of sense to federate the development effort and reap the benefits of code reuse, common tools etc.
Drawbacks are that it's nontrivial to set up and requires some rigid formalism in developer output that sometimes demands training and/or cultural change. But it's definitely something everyone should consider.
In my (currently internal, heavily LXC-utilizing but) infrastructure and OS neutral project, I am looking at specifically this sort of automation but for complex topologies of interdependent services, HA clustering layers, complex emulated network topologies (bonded links, multiple VLANs), etc. Plans are to include failure testing at the communications level (slow links, lossy links, cable snaps, switch failures, etc.) in addition to resource levels (disk, memory, etc.).
Outputs of a successful automated testing environment can include amazingly detailed information for capacity planning, automatically generated security policies (both for container-side, host-side and infrastructure-side deployment).
It's a fascinating area and one that is ripe for great change. Many people have needs here, the question is how to meet them at the intersection of current infrastructure and codebases, existing teams, business level concerns, varying hardware availability, etc. Both pre-commit and post-commit hooks are useful for different types of automation. IMHO LXC's blazing speed broadens significantly what can be tested with pre-commit.
If your dev environment is not different from prod, you're either insanely rich of your server setup is trivial.
I'm kind of surprised they didn't have Jenkins setup from the start; I'm also a bit taken aback that they don't use automated code reviews before accepting patches to their "deploy" branch. Even for a small project, it's not that hard to setup Jenkins+Gerrit to reject patches that break tests (or have to pass whatever other hurdles you want).
We've also had Jenkins set up for a long time now, we just used LXC to drastically improve our performance and scalability.
Here is an old blog post explaining some of how it all works: http://codeascraft.com/2011/04/20/divide-and-concur/
We also allow our developers to connect to a proxy to our production MySQL shards from their development environments in a read only mode. This allows them to leverage the large data sets that are quite hard to replicate in our development architecture. There is also a limited read/write mode that we are working on (with the proxy filtering dangerous queries). But all that is another blog post for another day.
We also do not use vagrant, opting for QEMU/KVM on physical hardware. The same tooling you saw in part 2 of my blog post also creates our development VMs as well.
"Run end-to-end, the 7,000 trunk tests would take about half an hour to execute. We split these tests up into subsets, and distribute those onto the 10 machines in our Jenkins cluster, where all the subsets can run concurrently..."
http://codeascraft.com/2011/04/20/divide-and-concur/
...so clearly running these tests on a single devs machine would be a bottleneck. The other use case is the dev env: in a previous blog they described how they're using their own internal cloud to run the dev vm's faster on dedicated hardware (with easy, one click provisioning):
http://codeascraft.com/2012/03/13/making-it-virtually-easy-t...
which makes sense. Why emulate prod running on dev's own boxes when you can pool the hardware and get better utilisation, & at the same time run them faster?
It is a good way to have faith in your ability to execute your stuff on new/unfamiliar/heterogenous environments, which can be valuable. People may be geographically distributed. People may wish to work offline. People may want a degree of control not available or even feasible on shared hardware resources.
So yes, don't rule it out, but still seems to make economic sense to share hardware.
Maybe worth looking at something like a small fusion-io / other PCI memory card