Edit: One box had really crazy, clever malware (backdoor IRC bot which was firmly detectable by NIDS (snort) and by remote nmap) which defeated local nmap, portmon and rootkitrevealer... and it was a 24x7 production oracle box (no HA or archive log mode master-slave repl) running the dining order, meals and inventory databases, so live cd / usb-hdd-specialized hdd dongle (better forensics) weren't possible.

0. If I had budget authority, I would've ordered something which could grab memory and disk images on a live system and sign-up multiple .edu researchers & symantec security group and equivalent shops under NDA to analyze them.

1. And I would've yanked all those janky (R)ILO (aka RMSA, aka DRAC) cards with their always outdated Linux / Java / PHP "wifi router"-like whatever embedded systems.

2. Finally, I would've spent some cash on honeynet setups and cc: to item 0.

Edit^2: Props to Josh Wieder for taking sounding the sec awareness alarm. I would only do active sec research on untrusted materials within a decent hypervisor's VM on a virtual desktop (VDI) which has "nonpersistence" on all storage, so it's clean on every power cycle.