If the Waffle House is closed, it's Time to Panic (2016)
fivethirtyeight.com
fivethirtyeight.com
https://en.wikipedia.org/wiki/October_2015_North_American_st...
The next morning, my girlfriend and I ventured out on foot to attend a yoga class. The yoga studio was closed, so we went to the Waffle House next door and got something to eat. They told us that their water supply had been compromised, but had made special arrangements so they could make coffee safely. We enjoyed a hearty breakfast.
We learned only later that some areas of town had gotten two feet of rain, that houses and businesses were underwater, that multiple dams had been breached, that we should boil all our tap water for a week, and that most people in town were without water entirely. My employer (the University of South Carolina) shut down for a week.
Indeed. Waffle House doesn't kid around.
I am incredibly impressed. They tend to look like glass boxes,* typically with at least two outer walls of large windows. One back wall is bricked in on the kitchen side and one end wall where the bathrooms are. The front end may have glass on three sides. This is where extra seating goes.
They typically have a galley kitchen with bar seating in the middle and booths to either side that are readily served by cooks and wait staff, plus that additional seating area towards the front (towards the main road, usually).
They have sort of a bad reputation as a dive restaurant, probably in part because the building looks so much like a glorified trailer and the prices are low, but the food is good and I have always loved the chain. My dad grew up on a farm, so I come from humble people who like down to earth places like this. Reading this article makes me feel really good about being a huge fan, in spite of the classist contempt some people have for the chain.
You would think that a place that (very often) looks like a glass trailer would not be the first thing open after a hurricane. So, I am astonished, but happily so.
* https://www.bing.com/images/search?q=waffle+house&go=Search&...
It was in that parking lot that some stranger tried to buy the family car from my parents. My parents bought it new and kept it for more than two decades. It had become a "classic" and was still in good shape.
Lots of good memories.
And I love their decor. It is practical and convenient. The layout is extremely efficient. If you know what you want, you can be served hot food in minutes.
I don't know why there aren't any other chains trying to imitate parts of their model.
>I don't know why there aren't any other chains trying to imitate parts of their model.
I've noticed this too. It seems like there's a split between "fast food" and "sit down" in the chain l world, bit no one else goes for the short order diner style. It's really a shame, as I think there's a big space for it.
Noodles and Co, Panera, Culvers, etc.
The other places are more upmarket, yes, because "greasy food for really cheap prices" is literally McDonalds. Anything between that and a real sit-down-and-order restaurant is called fast casual and you're going to pay a bit more for it.
Priceless. Is this yours, @Balgair?
Also, no downvotes guys, it was a simple question and totally in line with HN guidelines.
Anthony Bourdain enjoyed eating at one: https://www.youtube.com/watch?v=cX_kbIVxl_o
Now, natural disasters such as this one are much harder to plan for and deal with than the issues that could affect your average start-up. When you're moving and selling atoms the chain from raw materials to revenues is a very thin one with many weak links just waiting for the right conditions so they can break.
In contrast with that, in IT most disasters are man made and easily protected against. And yet, few companies do.
Its the weird crap that really gets your goat. I remember finding out the power main to our current place is located next to a T-section of road. Slide forward too far and hit the power main. Brilliant placement. Or when the primary, cannot fail contingency fails. A place I once worked at had a >$1 Million UPS (generator / batteries lasting weeks) for their data center. Semi hit the power main, and the UPS failed to turn on.
I actually got wrote up[1] as being "paranoid" in my backup strategy for a government grant we had in the 90s. The joke of it was that the site visitor who wrote me up was a project director at another site. That site experienced a flood and required extensive efforts to recreate their data. I was young, and a bit vengeful and might have made some comments during the next national meetup.
Sometimes, a lot of disaster recovery and contingency planning is politics above the level of the people who should know better.
1) They thought it was a problem during a site visit and we had to write a response that had to get approved.
Their presence, yes. But the frequency and intensity are hard to put any bounds on.
> I remember finding out the power main to our current place is located next to a T-section of road. Slide forward too far and hit the power main. Brilliant placement.
Something exactly like that happened with a gas line in Slupsk, Poland. The effects were incredible.
> A place I once worked at had a >$1 Million UPS (generator / batteries lasting weeks) for their data center. Semi hit the power main, and the UPS failed to turn on.
Just having a generator is not enough, you need batteries for the ride-through period and you need to do periodic tests. Maybe they did and it still didn't turn on?
I'd take being called paranoid as a badge of honor, better to be paranoid and up and running than down and searching the yellow pages for data recovery services.
[edit: about the site visit calling me paranoid] I was a bit ticked off, but I guess someone who didn't even do the basics, as we later found out, would not get the need.
When I ran a gas station in North America one of the first things I did was install a diesel generator. My reasoning was that with on average 50K liters of diesel in the underground tanks and the genset powering the pumps it should be good for most problems that we'd face. And in fact it was, which we managed to prove beyond any doubt when all of the North-West of the US and a good chunk of Canada went without power for a couple of days within a year of installing that genset.
Sometimes being prepared is a waste of money and time in retrospect, but it helps you sleep well, and sometimes it is the difference between a community continuing to function or people dying.
In the entire history of the Bell System prior to electronic switching, no central office was ever offline for more than 30 minutes for any reason other than a natural disaster.
Thanks for bringing it up!
Back when I was a medical tech, we lost a hard drive with patient data on it. I ran a chkdsk with no luck, and out of ideas, called a data recovery company. They stated it was $3-5k to attempt to recover, and asked if we'd done anything to the drive. "Just a chkdsk, nothing else". The guy on the other end of the phone was genuinely annoyed by this. "I wish you people wouldn't do that" delivered with that tone. Yes, this guy's position was that if you ever have a disk problem, your first port of call is to drop $3k+ at a data recovery place, don't even attempt to do some preliminary checking.
One day, the servers just vanish. We go into the server room to check. Someone had plugged all the machines into the same multi box / power extension cable and the fuse in it had popped.
Sometimes all your planning can be undone - especially once humans are involved :-)
Redundant power supplies on alternate phases is a pretty common element in designing any kind of server setup. So this wasn't a setup with a 'single point of failure', it was a setup with a very explicit single point of failure (several, in fact).
Someone linked to this excellent slide deck yesterday:
https://c3.nasa.gov/dashlink/static/media/other/ObservedFail...
Worth an hour of your time, not to make you paranoid but more to make it plain how hard this sort of planning really is.
Because it is said that if you are heading into the woods and worried you might get lost, take a length of fiber with you. If you get lost, simply bury the fiber in the ground. In five minutes a man in hi-viz with a backhoe will come along and dig it up.
So as a last desperate attempt, the person telling it basically sat watch inside the server room and noticed the door opening, a arm reaching in, unplugging a socket near the door, and plugging in a cable from the hallway.
The very next day security was switching the lock on that door.
That's not even remotely true. This is another one of those weird HN tunnel visions.
http://lakedillonfirerescue.org/wp-content/uploads/2013/03/w...
On a data center note, I have noticed a couple of data centers hooked to dams with farms as their neighboring land. I guess hydro is pretty reliable.
I live in a "green" area, nice low threat.
I'm also a volunteer firefighter. Last week I was at a brush fire in a semi urban area that got so bad so quickly that it prompted three choppers and the first time in this half of the state that a DC10 VLAT (very large aerial tanker) dropping retardant was deployed due to the threat to multiple residential and commercial structures, several of which were still lost.
In the lowest threat classification area.
It's why I'm extremely paranoid about dousing anything that was once on fire.
All it takes to start a fire is an idiot and a cigarette.
We think about risks.
We have contingency plans.
If I'm competing in a high growth industry against somebody who is busy planning for the worst, I'll move twice as fast and put them out of business before their plan can matter.
Lots of industries have clients that actually care about the robustness of their service providers as well; if in the future, once something invariably does go wrong, you have effectively just signed your own death warrant if you can't recover quickly enough.
Yes, maybe. Or you'll be put out of business because you did not have a plan.
You could attempt to justify your stance by saying that in the growth phase of the company nothing else matters but the reality is that you can't really retro-fit the attitude and set-up that your future company will require once you get there if you didn't do this from the beginning.
See, you can overdo anything, including security and contingency planning. But that isn't what this is about. It's about knowing that you are taking a conscious risk, knowing what the consequences could be and knowing that if those risks do materialize what you will do.
If some of those paths end with 'game over' then so be it. But there is no excuse for not knowing and it is exactly those things that you did not know about that will kill you.
So much good luck with moving twice as fast, headlong into a wall that you could have seen coming.
Or put another way, a good Plan B puts makes your Plan A possible in some situation that would otherwise mean failure.
Yes, you probably can, and more often than not your gamble will work. You'll move fast and disrupt and break stuff and odds are you can even say, up front, 'I'm moving fast to give you more without that boring planning stuff!' and may still be able to put the other guys out of business: depends on how well they handle it, and how hard you can push.
And then the natural disaster strikes and you're moving fast again, to make an insurance claim (or get out of town!) and the people who in turn gambled on you, are boned. Perhaps very catastrophically boned. And it's their fault, because let's say they were told up front that it was a gamble.
But people are dumb.
And you can put a lot of more responsible community oriented business-operators out of business before their planning matters, because people are dumb.
Depending on what you do (taco stands? Hospitals? Building dams and bridges?), your very Silicon Valley approach could be anything from colorful to criminal. I will hope the bridges I drive on are 'planned for the worst'. But there's only the inertia of social expectation, and whatever exists in terms of enforceable legislation and policing, to ensure that. And you can always violate legislation as part of your 'move faster' strategy, and expect to just deal with it somehow later. So that leaves only the inertia of social expectation.
Didn't matter; Still vested.
It is a pattern or process that repeats itself in everything from keeping web services online, stores stocked, casinos from being robbed, and even individuals from being dumped into poverty by events. I expect if there was a book "Everyday Prepping" it might actually sell pretty well :-)
My take away from the article however is that it is better to choose indicators that have a wide dynamic range :-).
The other thing is, it simply doesn't matter for most companies. They'll be out of commission for a week anyway because most of their employees are. There's no point have their online storefront online while the warehouse is have wheelbarrows of mud wheeled out.
As long as the data is backed up and recoverable, that's enough for most.
This comment doesn't have much purpose other than one of those small world moments.
[1] https://espnfivethirtyeight.files.wordpress.com/2016/12/prom...
Video: https://www.flickr.com/photos/jeremyjohnstone/4110293316/
still as with all organizations what also comes up is spinning off demand to other stores and warehouses. getting those outside the affected area ready to help those affected and so on.
the key though is communication and you must develop a well defined plan of how to communicate. this means from who, how often, the methods of delivery, and with email it can me using specific templates to make it readily apparent everyone is on board.
anecdotal, with family and friends of who volunteer for fema, there are just "constants" that they also use for knowing when things are getting better, some are restaurants and other are simply commodity foods and goods.
I would imagine that in so-called developed economies, there isn't anything except the "modern" supply chains. If a supply chain is using terms like "just-in-time" then there is no way of dealing with things when infrastructure goes down.
You guys are so screwed....
Don't even begin to compare Developed Western countries infrastructure and supply chains to anything in Thailand. To compare the two is laughable at best or shows your bias at worse.
I finally found one source of sustenance actually braving the weather that night: a White Castle, probably only because it was attached to a 24-hour gas station. At least I didn't have to go to bed hungry.
I have no idea whether there was a Waffle House in MI City, or where I might have found it if there were, but if all else fails... I guess there's always White Castle.
I guess it can't have been too bad