boden

I understand your perspective and will take your comments under consideration for future efforts or subsequent iterations of the existing benchmark.

However let me reiterate the intent of the existing work... The benchmark's purpose is to educate the community on the potential benefits of a LXC based technology (primarily from a Cloudy perspective) by providing a set of data points with full disclosure as to the means / methods used to perform tests and collect results. I expect any organization, group or individual considering actions as a result will perform their own validation to assert the technology choice is beneficial for their consumption prior to adoption. I believe this criteria is met in the current publication and therefore the results will stand on their own.

As to test configuration variations such as those stated above (disable KSM, boot by clone, etc.) -- I believe additional data sets collected via variations or other benchmarks are key to obtaining a well rounded understanding of the factors / benefits involved in this space and I encourage the community to create / publish such results. As an example I would like to gauge the impact of the various KVM guest cache modes and docker LXC storage providers which I believe are fair game to test and add as a variation to the existing benchmark.

Should you wish to iterate other configurations / requirements you believe are worth exploring I can make note of them for the next round of tests (time permitting). In addition I'm more than happy to document the instructions to setup and perform these tests so that others can run it themselves. I gladly welcome additional tests and data points from the broader community at large.

Agreed -- LXC as a technology is not a silver bullet and will not cover all use cases. However for those looking to do virt with Linux distro's IMHO the need to use traditional VMs becomes the edge case moving forward. I provided some details on the gaps and use cases for traditional VMs sections here: http://www.slideshare.net/BodenRussell/realizing-linux-conta...

All of the data including both sets of delete / reboot tests are provided in the benchmark presentation. How others decide to create headlines or side news is out of my control. From my POV the results contain all the data points and don't seek to mislead the reader.

Also keep in mind there are currently 23 pages of mysql images on docker index right now https://index.docker.io/search?q=mysql. Choosing an image which has "proper" bootstrapping is sometimes a trial and error process. This process is a bit more cut and dry with hypervisor / VM based images where you get a standard set of init processes out of the box.

RE KVM -- I agree KVM is impressive and the low memory overhead is likely a result of KSM (memory de-duplication) which is enabled out of the box on Ubuntu 12.04 with KVM 1.0.

Yes correct... As I mentioned below the tests are intended to show the benefits of an LXC based technology in the Cloud space. Docker was chosen b/c that's my preferred "LXC provider", but could have just as easily been libvirt-lxc, OpenVZ, etc.

The intent of the benchmark was to show benefits of LXC as a technology in the Cloudy space. It just so happened I chose docker as the LXC "provider" for these tests -- it could have just as easily been libvirt-lxc or something else.

Given OpenVZ is an LXC technology, comparing docker to OpenVZ would have been more of a comparison between different LXC providers. I agree this would be interesting, however not in line with the intent of the tests.. BTW -- I'm aware that OpenVZ is probably the most mature LXC solution out there.

As for docker vs Xen paravirt, that would be interesting as well and the information I presented clearly stated additional testing is warranted. I would encourage others to try such configurations and present results.