When Twitter was still in managed hosting and basically out of space to put new equipment we started an emergency "build a datacenter" project of which I was initially one of the only FTE's allocated too it. My plan was simple: I don't have bodies to throw at problems (we didn't even have data center techs and our facility was still under construction) so my plan was to make something that could effectively "self deploy" a functioning data center for us.

I locked myself in a really hot closet behind the cafeteria with several pieces of our soon to be purchased hardware and started coding. The whole thing ended up amazing and I am still proud of it to this day. =)

Some of my primary goals where simple: No manually edited configs, ever. If you swap a mother board there should be no mac address tracking, no serial number editing, etc. If we can detect it automatically we did and updated the machine database automatically. Hardware techs could simply walk out, swap a drive, and walk away and we would track that.

So we wrote a tool called vex that would export all the machines various bits and bobs. Anything we could access, serial numbers, mac addresses, etc. So once the machine was running we knew where it was, what it was doing, etc. The biggest challenge was getting it running in the first place. The plan was to use DHCP option 82 which includes port information when forwarding DHCP requests to an upstream server. Thus our dhcp server (written by an awesome coworker in like two days) was 100% stateless. It knew the IP range for the datacenter, the ip range for the rack, and the port that the request came from and could respond with a static IP every time. Swap the mother board and it wouldn't matter because the machine would still be plugged into the same port on the switch. Each machine would then request a tftp boot image from the server listed in the dhcp reply. This server was smart, it would look in the machine database to see what state the machine was in and where it needed to go "next" so to speak.

Step 1) Machine has never booted before. It would go into an identify utility that would allow the installer to verify that each machine was plugged into the right ports. The tech could watch the ipmi identify lights on the machine as it activated each in the rack in order and if any where out of order network cables needed fixed. From this point forward we knew that port 3 was the 3rd machine in the rack on the prod and ipmi networks. Once identified the machine would automatically reboot.

Step 2) The machine would run memtest86 to ensure that memory was valid. Sadly I didn't have time to make it automatically update with results and reboot so this step required manual verification when I left. A engineer would check IPMI and mark the test successful.

Step 3) The machine would run stressapptest to push the CPU and memory to the extreme. This allowed us to ensure that they would not overheat and that the rack could pull the load needed to get all the machines to 100%. Once done it would mark itself successful and reboot.

Step 4) The machine would run fio to ensure that every attached disk was in good state and fully functional. Again, it would then reboot automatically.

Step 5) The machine would install our default operating system assigning hostnames and such via parameters from the database.

Lastly once the machine booted into its "real os" for the first time it would setup puppet, install all the back packages, look into the machine database for its assignment and then install all the various bits ad bobs required for it to do its selected function (hadoop, web server, dhcp server, puppet server, etc)

Everything was driven by the machine database, and you could mark a machine as needing burn in and it would repeat the testing steps. Super useful if you suspect a machine is bad in some way but don't have time to debug. Once running vex would export all the details of the machine as well so we could track failure rates, hardware events, etc.

Best part is that the lab ran the same software as the datacenter so we could always test the various bits of this process to see what worked and what didn't. In fact I added several optional steps to do things like get performance profiles and build all kinds of cool graphs from tests on demo hardware.

I ended up bootstrapping three datacenters and all it required was an "Ark" which was just a single server with puppet, dhcp, tftp and audubon on it ready to setup all the other servers in the facility. During our initial power up we had a DHCP server doing something like 100-500 requests per second sustained over the course of like a half an hour (until the memory tests started)

Sorry, not sure why that memory jumped out at me but it seems you did something alike and it reminded me of how much fun it was to be involved in mine. =)