Hacker News Reader

Comment by mverwijs | Hacker News Reader

We did something similar at Optiver.

    * boot a custom live cd (a la Knoppix) over PXE
    * Live CD places node into database if it doesn't exist yet, 
    using dmidecode to find serialnumbers and such
    * Live Cd keeps querying database for instructions
    * Engineer adds a profile to the node in the database
    * Live Cd slices up disk to match the profile
    * Live Cd fetches a tarball of the base OS from an URL 
    and throws it on the metal. Runs grub setup. Reboots.

Tumblr did something like this with http://tumblr.github.io/collins/.

View on HN

Kind of amazing that the state of the art in this area hasn't changed in nearly 10 years. The marketing angle is funny as well - everything has to be a cloud now. I think I used to call what we built at Optiver a "private cloud" but "Metal Cloud" is nicely buzzwordy as well.

PS - Hi Marty, unknown nick here but I'm sure you can figure out who I am. :D

Also checkout https://github.com/Tumblr/genesis which does sort of the same thing as the PXE image GitHub writes about.

Actually there's heavy overlap in functionality if you include other Tumblr things with Collins like Phil, Genesis, and Configrr.

Collins - asset management

Phil - ipxe booting based on Collins state

Genesis - base hardware configuration (firmware, bios, raid, etc), burnin, kickstart

Configrr - state / config management

I did something sort of similar with Live CD images for booting hardware not on a PXE network in Cobbler. It could also register machines seen then first time by making system records for them.

https://github.com/cobbler/cobbler

What is interesting in this article in particular are the auto-firmware upgrade state transitions, which seem pretty neat.

The chat bot is I guess neat, but if a lot of people are using the cloud that could be a hard way to get status.

Hey cool so what I'm fiddling around with in iPXE isn't completely obsolete in the age of Docker and disposable VMs.

Well, no. But IMHO, iron is only do-able if you have enough of it. This I learned the hard way at my previous contract, where we only had a handful of servers and all of them production.

When you want to automate your complete infra, including rolling out hardware, you need hardware to develop and test on. Entropy of life will ensure that exactly that moment you need to reinstall that PostgreSQL slave from scratch, the PXE server is unreachable, or the server has a different diskcontroller, or an iLO certificate expired. Or something stupid.

Test your code. And for Ops this means: machines that are solely for the testing pleasure of Ops. No other function.

In a perfect world you'd have a Dev, Staging and Prod PXE server, but reality says you're not going to be able to get signoff to run that many.

But that is perfectly fine! That only means that you cannot have an automated install procedure that the company can rely on. There is really nothing wrong with a little manual labour at this stage. Do not spend weeks and weeks on automating this without having the environment to test your code.

Even those machines running Docker and disposable VMs still need to run on something.

I've got systems PXE booting to CoreOS where everything else runs in Docker containers (even the odd KVM VM).

Reply on news.ycombinator.com