Server room with seismic isolation floor in Japan earthquake disaster [video] | Hacker News Reader

Server room with seismic isolation floor in Japan earthquake disaster [video]

youtube.com

This is a great example of one of those investments that might feel like a waste, until it suddenly REALLY isn't.

I wish I had this video during my infosec & HA consulting days.

I live in seismicly boring Europe, so I have no idea what damage would be caused without this floor. Many broken hard drives? Or worse?

All drives broken at the very least, a 10k rpm drive doesn't like ordinary vibrations already, much less 40cm+ displacement in a fraction of a second.

Heavy servers and especially UPS battery rows would likely take the whole rack down/toppling over with them. Plus: most cabling is secured to the walls/floors with not very much slack, which means a LOT of damaged/torn cables and FUBAR'd ports.

All in all, if an earthquake of the scale in the video would hit a typical European DC, it's bye-bye and rebuild.

Forgive my uninformed question but I remember seeing a video many years ago where a (Samsung?) hard-drive had built-in protection from being dropped. The disk will stop spinning within a millisecond when it detects movement. They could survive a sudden drop onto a floor. I assumed this was a common feature of modern disk based hard-drives.

Could built-in protections like this in harddrives be a sufficient protection in an earthquake? What other moving parts are that at risk here? Assuming the servers themselves are locked in cages, I wouldn't expect them to crash to the floor.

Although the big difference here is it's not a sudden drop but a prolonged forceful shaking.

That keeps your data safe, but (and I'm guessing here) quite plausibly means filesystem writes may likely start failing, causing services on the affected server to start failing. If you need uptime during an earthquake, then that's not sufficient.

Why guess? From the article on the sensor:

"When the computer is stable, the drive operates normally again."

So unless you install a drive that has its own motion detection on board the computer will simply quickly 'park' the drive and will allow it to resume operation without any issues when it has landed.

to be fair, AlephGarden did say during an earthquake. Telecoms, Hospitals, etc, perhaps wouldn't be happy with being out for the time taken for an earthquake to blow over. Plus aftershocks when damage control is in progress.

They'd be a lot happier with being out for that time rather than having to replace a bunch of hardware.

A typical accelerometer can detect free falling and the hard drive can spin down before it hits the ground. I guess in that sense being shaken is worse than being dropped "suddenly".

Sudden motion detection [1] is built into many notebooks, and also allows you to use your MacBook as a lightsaber.

[1] https://en.wikipedia.org/wiki/Sudden_Motion_Sensor

And thinkpads.

I used it to add X11 autorotation to an X61 tablet. No spin animation, but it was a cool one-day hack nonetheless.

The fun part was that the orientation analyses was done from a shell script, and actually worked 100% of the time (unlike so many phones)! \o/

The insane part was that DRM/KMS on Linux is 1000% crazy and I had to patch the kernel to stop poking the backlight on every. rotation. event, grr. (I had to do it in the kernel - I basically killed DPMS - because X11 libdrm is utterly incomprehensible.)

These were fun, but if i remember rightly, Apple have largely phased the Sudden Motion Sensor out on all their laptop models that come SSD only, which is increasingly almost all of them. There were some ridiculous examples of using it to sort mail into folders in mail.app as well, by tilting the MacBook left and right.

Just guessing, but situations like these might be the place where it will make sense first to use server-side SSDs. While probably not there yet, it makes sense as the gap narrows that some day the cost of using SSDs is cheaper than building a whole floating/moving floor.

The SSDs address part of the problem. As noted by another post here, many rooms are build and structured with the assumption that nothing ever moves, so rather reasonably, you lay cable in a manner that has little flexibility, you mount things to floors/walls, and so on.

The data integrity is important, but the physical design and layout makes a rather big assumption that the infrastructure you're mounting stuff on will not move or vibrate.

Plus the DC itself might be running, but now you could be running it on a generator (no outside power); better hope you can get a fuel truck to it before the 1-week tank runs out (are there truckable roads outside now?). And connectivity - you might have line slack inside your building, but the fiber cables might be broken in several locations before they even get to next active element (which is also earthquake-proofed? And powered? For how long?); so perhaps you're down to the radio fallback (hope the antennas aren't misaligned too much, let alone that their masts are still upright); again, assuming that the other end of the link is operable.

In other words, preparing for vis major is far harder than it looks. Doable? Yes, if you're willing to go all the way to military-grade resilient comms...for a military-grade budget. Far easier to failover to a DC in a different zone.

> All drives broken at the very least, a 10k rpm drive doesn't like ordinary vibrations already, much less 40cm+ displacement in a fraction of a second.

Do commercial server drives not have the same anti-shock tech as laptop drives?

Drops are easier because there's a gap of free fall before the actual hit so there's time to sense and park the heads. Earthquakes are more instantaneous where the shaking is the first warning the protection can react to.

Which gets you into that boring part of actuary calculations. In seismically inactive parts, "bye-bye and rebuild (while the cluster is running from a geographically remote datacenter, and only a slight increase in latency indicates this)" could be an order of magnitude cheaper, and thus preferable. In other words, downtime is not replaceable, hardware is.

I remember reading on http://reddit.com/r/talesfromtechsupport some time ago (I unfortunately don't remember what thread) about someone who decided to move their server rack. Said rack had a UPS, but no wheels.

The 10k HDDs in the blades didn't really like it...

I remember us having some server issues a few years back when they were blowing up rocks for the foundation for a new mall a few 100 meters away in Stockholm. You never know.

Apple's new giant circle can move 4.5 feet! https://www.wired.com/2017/05/apple-park-new-silicon-valley-...

One of cap Gemini's offices in the UK is built on springs and has a dry moat. Though I think that was for bomb attacks not earth quakes.

The moat might also be useful when besieged.

I can't help but chuckle at a writer using the term "insanely great" to describe the building. It lacks a certain potency in the context of a book's tagline! Perhaps it's a symptom of the exagerated language pop culture uses for average things diluting the power of other possible words.

The article was however, "fairly excellent".

"Insanely great" is a Steve Jobs-ism, so in the context of Apple it's usually a reference to that.

Oh I see, that makes sense then. Thank you for the clarification!

This feature is built into a lot of critical infra there, like airports. As I'm sure you know, the more common (and costly) it is, the better investment it is.

But apparently no protection against power surges /s

http://fortune.com/2017/05/31/british-airways-it-collapse-se...

Poor sysadmins! They have to wear full 3 pieces suits on the DC Floor!

Besides that, the video/tech is really cool.

At least it's a nice cool temperature.

Depends which aisle you're in :)

That day, I wouldn't call them poor. Probably the safest place out there.

I visited a couple of DCs in Tokyo, and the tours systematically feature a look at the anti-seismic system. It's actually pretty unimpressive : basically the seismic-protected part of the building is mounted on big rubber dampers, with some huge pistons thrown in for active attenuation. In taller buildings (I visited a DC with around 20 stories IIRC) the stories are not rigidly connected together, so that instead of swaying (and possibly toppling) during an earthquake, the building just kind of wobbles.

Very effective though, as the video shows. My visits were post-2011, and each DC had a record of the building's movement during the big earthquake; max amplitude on the seismic-protected part was a couple of centimeters, vs 50 cms or more for the rest of the building.

Unimpressive?!

Well, I am quite impressed.

Just to clarify: I meant that the equipment itself is unimpressive (mundane would be a better term I guess) but the results are definitely impressive :D

Very impressive, actually.

More impressive are the guys walking around during all that swaying as if they were on coffee brake.

whole building isolation/damper system vs no dampening comparison:

https://www.youtube.com/watch?v=kzVvd4Dk6sw

This is another neat demo: https://www.youtube.com/watch?v=xp2pGxFzrzI

> In taller buildings (I visited a DC with around 20 stories IIRC) the stories are not rigidly connected together, so that instead of swaying (and possibly toppling) during an earthquake, the building just kind of wobbles.

This is the reason why Japanese pagodas have survived for so long.

https://youtu.be/0tFWn_e71qc?t=2m45s

It's not a DC, but Taipei 101 has a mass damper, and there's a video of it swaying during an earthquake. Really impressive how much that things moves!

https://www.youtube.com/watch?v=NYSgd1XSZXc

Any video editors got a few minutes?

There's an open video stabilization request here: https://www.reddit.com/r/ImageStabilization/comments/6e1mgj/...

So far there's https://streamable.com/lrkhg which sorta gets partway there.

I've been in a datacenter in Tokyo where the floors were not seismically isolated, but the rack cabinets were made to sway. It's hard to tell many floors up that an earthquake is in progress, but you can see it when you see the racks swaying!

I visited the Dreamworks DC in SOCAL back in 2005 and they had per rack isolation plates as you remember. They also had a 25' long tape drive on one huge isolation plate. Those plates basically are on a central piston and rotate in a circular pattern. Seems a lot cheaper than proofing the entire floor.

I wonder if it is disorienting standing or walking on the movement isolating floor.

There's a (thought?) experiment you can do where you put someone inside a fake room built like an overturned box, and then jerk the walls in a random direction while the floor stay still. You get disoriented and fall over. Or the more common experience where you're on a stopped train and the train next to you starts to move, and it takes you a minute to figure out which of you is actually moving.

I'd have to imagine standing on the isolated floor while the rest of the building moves would be similarly confusing.

or like the spinning tunnel effect, where the walls are spinning around you and you have difficulty walking straight because your brain thinks you're falling over

https://en.wikipedia.org/wiki/Spinning_tunnel

The guys in that video look to me like they are experiencing some vertigo from the room shifting. They keep grabbing things to steady themselves.

The apparent motion of the floor makes running along the edge of the gap, as one guy does towards the end, look particularly risky.

Category "Auto and Vehicles" ? It is interesting to note that the people on the seismically isolated floor are not swaying with the video (it is explained that the camera is mounted to the non-isolated part of the building so it appears the server room floor is moving when instead it is the building that is moving).

Most of the data centers I've looked at in the Bay Area just bolt the racks to the floor and are done with it assuming, I presume, that it is the shifting on the floor that damages the hardware.

Probably more an issue of shifting racks damaging the employees. Bolt those puppies down and nobody gets crushed to death.

That earthquake goes on for way longer than I would have expected.

The duration and intensity of an Earthquake are strongly correlated. You can even use it to estimate how strong a quake is and how far away you might be.

The Sendai earthquake in 2011 was 6 minutes long. There's a story which I now cannot find about a seismic conference being held in Japan some distance from the epicenter. They felt only moderate shaking, but were immediately aware of the severity of the quake due to the duration.

Relevant portion starts right at the top: http://www.newyorker.com/magazine/2015/07/20/the-really-big-...

What is amazing is how cool are the people in the video. Instead of running out screaming they calmly do their job.

There are an astoundingly high number of quakes in Japan that are strong enough that you can feel. When I was in Tokyo years ago, I felt one every four weeks on average.

You quickly come to a different relationship with the world when it refuses to stay still that often.

I sure hope that being among the racks is also the safest option for them, and that they are not risking anything just to keep these servers running.

Commercial racks and datacenter products have optional features that can be utilised in earthquake prone environments. What you see in the video is not only those rack features but also structure of the building itself (both are important and neither one is sufficient for certain areas).

For example see: https://www.ibm.com/support/knowledgecenter/STXN8P/com.ibm.s...

Without employing these techniques, you'd get something similar to this: https://i.imgur.com/Sb2M5Qo.jpg

Can a multimedia person make the video fixed on the servers instead?

I guess a similar level of movement isolation would probably be required for all those repurposed oil rigs and maritime-oriented machine rooms that anticipate taking advatage of sea water as server coolant.

"Techno Mind Corporation".

Think I saw them play in Berlin once.

Reply on news.ycombinator.com