Running MemSQL’s 107 Node Test Infrastructure on CoreOS [video] | Hacker News Reader

Running MemSQL’s 107 Node Test Infrastructure on CoreOS [video]

blog.memsql.com

33 pointscarlsverre9 comments

"At a recent CoreOS meetup, I gave a presentation on the design and implementation of Psyduck, which is the name for MemSQL’s test infrastructure."

So does the test infrastructure get exponentially more powerful when under pressure, or what?

Interestingly enough, in some ways it does. For example, during our nightly stress runs the fact that the system is under so much pressure helps induce race conditions that are otherwise pretty much impossible to reproduce.

Awesome! I wish I had the opportunity to work with a large scale docker/coreos deployment like this. I really perked up when you started talking about PXE booting, I wanted to see what you guys were doing to manage updates.

I'm trying to make automated updates when using PXE[0].

When installed to disk, CoreOS's update manager automatically polls for updates and downloads them in the background. On next reboot, the new version is shifted into place. This is unfortunately unavailable when booting with PXE.

Working on large server deployments is a blast! And CoreOS makes it even more fun via its read-only focus.

In our system we don't do automated updates since we don't need that functionality. In order to send out an update I download the image that I want to deploy, patch it with CPIO (with my custom initialization code) and then register it with DNSMasq (put it in a directory). At that point any machine that reboots will start up with the new image rather than the old one. I usually test one/two machines and then if they look good a script reboots the rest.

If you wanted to do automated deployments, you could basically do the same thing provided you have a server to manage the process. The master server would simply poll CoreOS for new releases, and then do the same steps as above to roll out the image. Incremental updates are trivial as well since you are just rebooting the machines.

One thing to consider is running CoreOS out of memory rather than installing it to disk. I find this is one of the best features of CoreOS since it makes it even harder to think of a machine as any more than a temporary computing resource.

Hope that helps! Good luck!

My thought was to have CoreOS run the systemd cron job for update_engine even when booting off of PXE, but to proxy[0] through corepxe (the jawn I'm trying to write). This way I can MITM the request and make the request on behalf of the host from corepxe, and take advantage of how omaha[1] works by returning the appropriate status code and downloading in the background.

[0]https://coreos.com/docs/cluster-management/setup/update-stra... [1]https://code.google.com/p/omaha/

That seems reasonable, however I am not super familiar with how update_engine works. Likely your solution will work provided you setup your image to install itself to disk and boot from there.

Can't you just PXE boot a bootloader (e.g. grub) rather than a kernel, and then have the bootloader load the kernel as normal?

The boot loader would then need network drivers to download the correct kernel. Off the top of my head I don't know how much GRUB supports. iPXE may be useful for this sort of setup.

At some point we would like to provide a update client for managing a PXE server but that isn't a high priority right now so it has yet to get past the "wouldn't it be nice" conversation stage.

Reply on news.ycombinator.com