Shouting in the Datacenter (2008) [video] | Hacker News Reader

Shouting in the Datacenter (2008) [video]

youtube.com

437 pointsmmastrac78 comments

Pretty funny to see this video at the top of HN! My kids still think I'm an idiot for not having somehow parlayed the enduring popularity of this video into being a YouTuber -- despite my protestations that I was merely the videographer here. If anyone is curious about the video's origins, I had a fun conversation about it with Ben Sigelman a few years ago.[0]

[0] https://www.youtube.com/watch?v=_IYzD_NR0W4

Will oxide racks have a mode where you can make them slow down when you shout at them?

that was really funny

Make sure to read the first few YouTube comments for the full humor

Best comment IMO

> Even soullsess machine cant work properly when it`s being shouted at

I'll be sure to check it out

28:45 in the video. https://youtu.be/_IYzD_NR0W4?t=1725

Heard about this on the Changelog podcast recently, great interview!

I'm with them on this issue! Go for it. You have a platform, man. :-D

In case anyone needs to know, the shouter in question is none other than Brendan Gregg (of flame graph, eBPF, and Systems Performance fame). And the guy who took the video is Bryan Cantrill of Oxide Computer, DTrace, and Joyent.

I think Brendan is still peeved that this video is his most viewed video on Youtube even after all this time.

Yes, whilst he has given so many other valuable talks around systems profiling, etc..

I firmly believe that this [1] still should be hanging in every system administrators bedroom around.

https://www.brendangregg.com/Perf/linux_observability_tools....

I do have it hanging in my office rather than my bedroom. I wouldn't recommend putting it in the bedroom unless one's spouse is also a sysadmin.

Maybe he should instrument his talks to see how much shouting there is in them, and check how that correlates with view count.

This is a flame graph I'd be interested in seeing from him on April Fool's Day

I think mcelog/rasdaemon deserve a mention there in the RAM section?

He shouldn't be too upset, views and video quality only have a very rough correlation if at all. It shouldn't be surprising that his most accessible video is his most watched one.

HN readers may also recognize Byran Cantrill from his famous rant about Oracle, which is frequently cited here: https://youtu.be/-zRN7XLCRhc?t=1980

>Don't make the mistake of anthropomorphizing Larry Ellison

Absolutely incredible!

I have trained myself so that every time I see "Oracle", I mentally translate it as "Larry Ellison needs a bigger yacht".

I keep picturing Larry pushing a lawn mower.

For Spanish speaking HNers: try reading Oracle in reverse :)

It'd be el caro (the expensive one), in the same line as the yatch thing.

I hope to one day have the terrible misfortune of being acquired by Oracle.

Brilliant. Thanks.

FWIW I did a "making of" video soon after, in 2009: https://www.youtube.com/watch?v=lMPozJFC8g0 . Only bit I missed discussing there was that I did try to download freeware frequency generators wondering if there was some "rust note" of HDD harmonics, but those I downloaded didn't work and out of frustration I decided to start yelling instead.

Thanks for sharing!

Oh man. Once, when I was a young padawan doing maintenance by myself on one of our three racks in a colocation DC, I dropped a 1U server from about 2m/6ft high. It was scary as hell. While trying to grab it I hit the off button on one of the other servers, causing an outage for about 50 customers. What's more, it fell on the floor and just missed the SAN with the spinning disk array at the bottom of the rack. Everything was fine, including the fallen server, but shit, that was stupid. The vibration alone could have caused a lot of data corruption.

I’ve had old drives that a bit of a whack on the side have given them the kick to start working again (such that you can get data off)

Old grease, you probably broke the sticking just enough for the motor to overcome it and then after a bit of turning the grease warms up enough that it will liquify.

There's an interesting little paper 'Acoustic Denial of Service Attacks on HDDs' where they use a function generator to find resonant frequencies which cause issues with spinning disks

There's also a paper Hard Drive of Hearing: Disks that Eavesdrop with a Synthesized Microphone. An APT attacker can potentially reflash the firmware on an HDD and using it as a microphone for eavesdropping.

https://andrewkwong.org/docs/Kwong-HDDphone-IEEE-SP-2019.pdf

Janet Jackson's song Rhythm Nation caused certain 5400 rpm HDD's in laptops to crash:

https://devblogs.microsoft.com/oldnewthing/20220816-00/?p=10...

Yeah I noticed this phenomenon already, every times I'm shouting at my collaborators they start crying and procrastinating, classic.

I have a colleague who was on the on-call rotation for some of our systems hosted in a datacenter our company owned. During this rotation he got unlucky in that the datacenter detected fire and decided to disperse gas that displaces oxygen (I think it was Halon gas?). Those systems are no joke - you have a short warning before you are supposed to put on a mask or get out of the server room otherwise you will just suffocate.

Anyways, no fire damage was detected but oh boy... he had to run around various tech stores left and right to buy about 50-100 enterprise grade spinning disks before returning to the site. The fire suppression system somehow broke most of the hard drives in the server room.

This was more than a decade ago, however. Most of the disks are replaced with SSDs and so far they seem to be much more resilient than the spinning ones.

I was once told by a datacenter operator that this is due to the pressure change when they pump the gas in and that we should plan on replacing all HDDs should the fire alarm go off.

...and I think it was Argon...

Thanks for jogging my memory!

It was indeed Argon, not Halon, and the pressure issues that broke HDDs.

And the other comment made me google noise vs. pressure....

Looks like Siemens did an investigation and tested various parameters (pressure, noise, vibration) independently.

They point towards noise, not pressure. Really interesting as I am _sure_ I was told it is pressure. But that might be 100% wrong!

https://virz.de/wp-content/uploads/2020/05/2015-White-Paper-...

Noise really is just pressure, with a high frequency time component.

This is particularly noticeable with over-ear noise cancelling headphones.

The feeling of anti-sound is quite distinctive.

It does feel like ‘pressure’ on your ears/inside your head. Not very comfy actually

My partner worked in a Research School of Chemistry.

On the top floor of the building a tip-over failure replacing an argon bottle broke the valve off the end and emptied the compressed lot of Argon in short order.

Argon being colourless, ordourless, heavier than air = whole building "anti-suffocation" evacuation. But no fire problem!

> The fire suppression system somehow broke most of the hard drives in the server room

Possibly another case of loud noise when the gas is released - see e.g.

https://www.theregister.com/2018/04/26/decibels_destroy_disk...

https://www.bbc.co.uk/news/technology-37337868

spinning hard drives of the pre-helium-filled generation are not in fact totally sealed.

if you disassemble one you will find the tiny air filter module which covers a hole in the casing for equalizing pressure with the outside air.

I wouldn't be surprised if the fact that argon is heavier caused the failure. Argon gets into the drives, assisted by the larger pressure outside of the drive at the time of the release, and it wouldn't come out within a reasonable time even when the pressure came back to normal, since there's not much capability for venting gasses out of the hard drive. And since argon is significantly heavier, it causes larger air drag on the mechanical parts of the drive, eventually making the head driving coils too weak to operate at the needed speeds.

> This was more than a decade ago

He’d be even more on an inclined plane, wrapped helically around an axis nowadays as there probably isn’t even one never mind multiple local places to buy that amount of enterprise drives.

Might be a micro center if you’re lucky.

The shockwave from a sudden release like that can take out the better part of a raid array in a matter of moments. I never understood why they have to release it all at once instead of gradually because the number of times that I've seen accidental releases dwarfs the number of real firefighting incidents.

Same effect mentioned here: https://arstechnica.com/gadgets/2022/08/janet-jacksons-rhyth...

and here: https://devblogs.microsoft.com/oldnewthing/20220816-00/?p=10...

I recall this fondly when it came out. I had a few racks of Sun servers at that time...and anyone I showed the video too were convinced it was bullshit. April Fools anyone? However, Brendan and Bryan know their onions...and all the naysayers had to pause and think a bit. It too crazy to believe...and too crazy to not believe. I miss this stuff.

Edit: This was done when everyone ran spinning media. I wonder how modern SSDs hold up?

This works. I tried this on a Sun ZFS7320 under synthetic load to utilize all disks.

We all laughed to the video, too.

I did it on our fishworks box as well, before we put data on it. That could have been an amazing product if not for Oracle.

I wouldn't want to spend too much time in there without ear protection! Is it common for data centers to be this loud?

Yeah, the fan noise is that loud. You should use hearing protection in there, even when staying for a short amount of time. Though I have heard about some special DCs that use water cooling and as such can defer the noise away from the racks.

Sure -- especially in the hot aisles it can be hard to have conversations. I doubt it's much worse than a nightclub though.

Eh? Certainly back when all the disks spun. That and the server fans and the A/C make quite the racket.

I've been in my share of datacenters and most of 'em are at least a little bit quieter now. Still loud, but not to the extent that you have to shout constantly.

It varies. Once I've been at a customer's data center with raised floors some ten years ago. It felt like an hurricane when you removed one of the tiles. Loud and cold (ear protection and wind breaker mandatory). Hard to communicate there, harder still to think. Not a nice place to be. Keep it in mind when choosing your career.

I got a "3M Peltor X5A" a couple years back as a "thinking cap": put on to think in thought-hostile environments. Works like a charm. Probably best 30 bucks I ever spent.

For comms use a wind-proof mic on a headset boom and earphones under the X5A.

We do Vibration Testing where I work (think satellite parts, simulating launch, etc). We moved to SSDs a long time ago because some of our test gear would shutdown at certain frequencies/levels during vib tests.

Reminds me of the mummy shouting thing: https://www.youtube.com/watch?v=G8qhrURrQbI

Fus ro data loss!

Underappreciated comment

Does this mean you could improve datacenter performance and lifespan by installing acoustic tiles and such?

This reminds me of an old favourite video, also in a Data Centre, also low-tier virality: https://www.youtube.com/watch?v=LeFtwtvy4Wc

Any other Data Centre classics I should watch?

Years ago someone had an idea to put electron microscope in a basement of a uni building next to a busy, city-center street in Cracow, Poland, only to be able to use it during nights due to unstable images. Fun times.

What a classic! One of the best feature demo videos of all time.

I got all solid state drives so I can scream non-stop

Always makes me laugh ! We need more funny content on the top page!

Does yelling at your disks shorten their lifespan?

Only one way to find out...

weeks later

"Yeah, we had to let him go. Too burned out. Caught him screaming at the SAN."

End up like this guy: https://m.youtube.com/watch?v=ghv31AUufNk

Its a Solaris machine...not surprised

Can you point me to your references that indicate HDDs running Solaris are the only ones prone to loud noises disrupting their mechanical nature?

It's a joke

Old man yells in cloud.

Who's laughing now?

Amazon

Lol, this was brought up before... at least one time:

https://hn.algolia.com/?dateRange=all&query=tDacjrSCeq4&sort...

> Please don't complain that a submission is inappropriate. If a story is spam or off-topic, flag it. Don't feed egregious comments by replying; flag them instead. If you flag, please don't also comment that you did.

https://news.ycombinator.com/newsguidelines.html

Reply on news.ycombinator.com