Hacker News Reader

Comment by boden | Hacker News Reader

All of the data including both sets of delete / reboot tests are provided in the benchmark presentation. How others decide to create headlines or side news is out of my control. From my POV the results contain all the data points and don't seek to mislead the reader.

Also keep in mind there are currently 23 pages of mysql images on docker index right now https://index.docker.io/search?q=mysql. Choosing an image which has "proper" bootstrapping is sometimes a trial and error process. This process is a bit more cut and dry with hypervisor / VM based images where you get a standard set of init processes out of the box.

RE KVM -- I agree KVM is impressive and the low memory overhead is likely a result of KSM (memory de-duplication) which is enabled out of the box on Ubuntu 12.04 with KVM 1.0.

View on HN

The problem is not that you didn't publish the data. It is that, when collecting it, you chose to fix/re-run some experiments, but not others. The ones you chose to re-run were (presumably) based on your pre-conceived notions of what things "should be". The slippery slope here is that, if you had different notions, perhaps you would have tried to fix different experiments. Perhaps KVM would have been faster if you believed it should be faster and so had used one of the boot-by-cloning-instance patches. Perhaps the memory usage would have been higher if you thought it should be and so turned off memory sharing (and then maybe the CPU load would have been much lower).

I appreciate the transparency in admitting that you re-ran the test, and appreciate anyone that tries to provide data for OpenStack. But you shouldn't claim that you're just publishing numbers for people to interpret as they will, when interpretation entered into the production of those results. It's just not valid experimental practice.

I understand your perspective and will take your comments under consideration for future efforts or subsequent iterations of the existing benchmark.

However let me reiterate the intent of the existing work... The benchmark's purpose is to educate the community on the potential benefits of a LXC based technology (primarily from a Cloudy perspective) by providing a set of data points with full disclosure as to the means / methods used to perform tests and collect results. I expect any organization, group or individual considering actions as a result will perform their own validation to assert the technology choice is beneficial for their consumption prior to adoption. I believe this criteria is met in the current publication and therefore the results will stand on their own.

As to test configuration variations such as those stated above (disable KSM, boot by clone, etc.) -- I believe additional data sets collected via variations or other benchmarks are key to obtaining a well rounded understanding of the factors / benefits involved in this space and I encourage the community to create / publish such results. As an example I would like to gauge the impact of the various KVM guest cache modes and docker LXC storage providers which I believe are fair game to test and add as a variation to the existing benchmark.

Should you wish to iterate other configurations / requirements you believe are worth exploring I can make note of them for the next round of tests (time permitting). In addition I'm more than happy to document the instructions to setup and perform these tests so that others can run it themselves. I gladly welcome additional tests and data points from the broader community at large.

Reply on news.ycombinator.com