Hacker News Reader

Comment by teraflop | Hacker News Reader

The paper links to this entertaining YouTube video from 2009, in which someone demonstrates the effect by literally shouting at a rack of servers: https://www.youtube.com/watch?v=tDacjrSCeq4

(EDIT: replaced the link I found via YouTube's search with the original, thanks robotmlg!)

View on HN

That's a reposted video, the original is from Bryan Cantrill (formerly of Sun, now CTO of Joyent): https://www.youtube.com/watch?v=tDacjrSCeq4

Wow.. so, how much interference is caused by the ambient noise of the server room itself? Is there some "critical threshold" for noise beyond which you'd have to install sound dampeners or smth?

I've had a case that caused disk performance to drop because the memory correctable errors rose above a server BMC threshold.

Turned out that the BMC beeped the internal speaker which vibrated the chassis at the right harmonic frequency that matches the HDD sensitivity spot and reduced performance. After that we disconnected the speaker on all servers.

The overall nose may impact the HDD performance but mostly if the vibration is at the right frequencies. Some are a lot more important than others.

that someone is Brendan Gregg! He's awesome! http://www.brendangregg.com/

Holy shit that's one hell of a demonstration!

I wonder if somewhere out there there's a sysadmin pulling their hair out trying to figure out why their I/O shits the bed every now and then.

Several jobs ago, we were in the data center and doing this came up. We had an idle disk shelf with which to test, so we started up a bulk IO operation (`dd` or something; I don't remember what, specifically), and watched the dip in the IO rate correlate perfectly with someone crouching next to the shelf and screaming at it.

I remembered to read somewhere that the fire suppression in DC could kill HDDs just in the same way (if they use gaseous agents). Never had fire drills in our DC to "confirm" that though.

Ive had that happen. Only killed 1 or 2 disks but it knocked a lot more disks offline requiring a reboot. Not bad out of 2 racks I guess.

In that case the valve holding back the gas failed so it was an unexpected release. Would not have wanted to be in there at the time - no warning!

If shouting works... couldn't the typical LOUD data center noises of other machines and AC, also cause disk latency?

I know vibration affects hard drives, but (AFAIK) data center drives are mounted on anti-vibration mounts for exactly that reason. But sound? Goodness!

Reply on news.ycombinator.com