Went like this: Guy who shows around the new datacenter/ops guy demonstrated how the emergency power off works by lifting the protection plate. Protection plate unhinges suddenly and droppes onto emergency power off button. Hilarity ensues.
Went like this: Guy who shows around the new datacenter/ops guy demonstrated how the emergency power off works by lifting the protection plate. Protection plate unhinges suddenly and droppes onto emergency power off button. Hilarity ensues.
Good thing we deployed those EMCs in pairs. But then curiosity gets the better of a DC ops guy wondering how the plate got jammed. So he punches the plate on the 2nd EMC causing it to similarly jam and power-off. Doh.
When we told EMC about this flaw in the design, they deployed a fix to the production line - a rubber stopper under the plate next to the switch to protect the switch from the protection plate.
Confused the heck out of us when we were trying to figure out why some of our servers went on UPS power randomly at night. Turns out we'd get the notifications on nights the cleaning crew decided to flick the lights off then on again.
There's a plate on the switch now, praise Eris.
Had another instance where a pipe ran from the floor above right over the UPS. Construction workers on the first floor decided to poor some unusued paint into it (figured it was a drain pipe) and of course, we lost power again.