Hacker News Reader

Comment by mysterypie | Hacker News Reader

When the World Trade Center collapsed during 9/11, those buildings must have had hundreds or thousands of safes. Many high-quality safes must have survived the fire and impact. Did anyone try to recover them or were they hauled off in the cleanup (100,000 trucks full of debris)?

View on HN

A colleague of mine worked for EMC in the 90s and early 2000s, and has some insane stories around 9/11. They drove him down to New York later that week, where he joined a team of federal reserve, SEC, DTCC, and army combat engineers whose job was to locate at ground zero the remains of a few EMC storage arrays and pull the drives. These frames had transaction records on them that had to be recovered in order to open the markets on the 17th. The issue was that the primary and secondary replicas were in data centers in building 1 and 2. No one had ever considered that both buildings could be destroyed simultaneously. Shortly thereafter, three-site mirroring with an asynchronous leg out-of-metro became a mandatory standard for all mission critical banking data.

Wow, that makes a great disaster recovery warning story

How successful were they? Was all of the data intact? If not, how did they deal with incomplete records for the stock market opening?

If I remember correctly all they needed was the last 30 seconds or so of missing data, and they were able to recover it off the drives in the symmetrix lab up in Hopkinton.

EMC sales types told us that the EMC method of using "vault" disks--disks that cached writes for the rest of the SAN--saved the bacon of various companies that lost power in lower Manhattan that day.

Not physical safes, but there's a fascinating story about Cantor Fitzgerald, a financial services firm that lost over 600 employees in the attack, calling in Microsoft security experts to break into hundreds of employee accounts they no longer had access to. Client money was tied up in these accounts and they needed to regain access before the stock market opened up again.

To help in this task, they had to call the family members of still-missing employees and ask for personal information that might be included in passwords:

  "What is your wedding anniversary? Tell me again where he 
  went for undergrad? You guys have a dog, don’t you? What’s 
  her name? You have two children. Can you give me their birth 
  dates?”

https://www.nytimes.com/2014/11/19/magazine/the-secret-life-...

>call the family members of still-missing employees and ask for personal information

When my cousin passed very unexpectedly, this is exactly what I got called in to do... was easy enough for me to take some educated guesses and figure it out, I can't imagine that being suitable for a large firm to handle their password issues.

Multiple brokerages, most famously Merrill Lynch, had both, primary datacenters and safes containing daily backup tapes in the WTC. There were unknown dependencies and backup systems that had been presumed to be independent did not work. Some secondary datacenters were in adjacent buildings also damaged by the attack. It had been assumed, given the extra protections put in place after the prior bombing, that the WTC was as safe as it gets so some firms had primary and secondary datacenters in different towers or different floors of the WTC. Multiple safes were plucked out and databases of customer positions were restored during the week the stock markets were closed. Getting these safes and the tapes out was given priority despite everything else going on because if the databases had not been brought back on line, there would have been absolute chaos in the markets.

Immediately after the attack, exchanges moved their datacenter operations to sites in NJ about 80 miles apart and both exchanges and large brokerages setup backup datacenters in Delaware, far enough away to be unaffected by a thermonuclear blast hitting New York.

I believe, prior to 2001 only JP Morgan (pre-Chase acquisition JP Morgan) had backup datacenters in Delaware but there may have been one or two others.

Brokerages had backup DC's in Journal Square(IIRC) long before 9/11. It was apparently a favored location as you could draw power from 3 'independent' grids. After 9/11, a number of them didn't bother transitioning back to Wall St. I worked on Wall St. during the 90s, and was really shocked how much of a ghost town it had become even 3 years after the attack.

Are there really safes in existence that would survive the fire that melted the building's steel infrastructure and the building collapse?

Presumably the entire building and its contents did not melt. There were probably many kicking around in the rubble.

Not while in the jet fuel, but that was localized. A tiny number of people survived the collapse while inside the WTC. https://www.history.com/news/world-trade-center-stairwell-de.... Miraculously, their small group ended up being the only survivors within the stairwells of the north tower after it collapsed, somehow emerging from a pocket of rubble.

Presumably the contents of various safes similarly survived.

oh duh, that makes sense.

The ones that could are probably too heavy to be 50 stories up.

Figure the typical sporting-goods store gun safe weighs about 350 lbs (159 kg) and isn't really all that secure (a sawzall with a metal blade can open them). They don't do much better if you drop them from "only" 200 feet up (61 meters):

https://www.youtube.com/watch?v=xQFppQZRdqY

my sporting goods safe is the size of a full size fridge and weighs 1400 lbs

That’s not a ‘typical’ gun safe. It’s not uncommon, but it’s definitely it ‘typical’.

When I worked on Wall St. in the 90s, there were persistent stories about Iron Mountain having vaults in the sub-basements of WTC - mostly in relation to mag tape storage. Never had the opportunity to see them myself, so it might have been bull.

I thought there was literally tons of gold stored in the WTC. Now I'm going to have to go look up the stories.

Reply on news.ycombinator.com