Whiteboards used after Gatwick flight information screens fail | Hacker News Reader

Whiteboards used after Gatwick flight information screens fail

bbc.com

154 pointsgpresot111 comments

Seems like a pretty big success to me. They had a contingency plan and it was implemented successfully. No cancelled flights and minimal delays - good job.

Presumably it helps that many (most?) travellers will be using the airline's apps these days, which provide info such as boarding times and gate updates.

In my experience, all the apps (airport, airline, travel apps, even Apple Wallet e-tickets) all have unreliable or heavily delayed gate information. Does anyone really rely on them?

Going to the airline's site and looking up the flight status has often resulted in a better update on delays, gate changes, and etc. for me than even the airport's status displays.

Info viewed directly in the Apple Wallet screen is delayed, for sure. That applies to all apps - the tickets/"cards" don't seem to refresh that often. But what you see inside the app should generally be up to date, although I guess not all apps are equal.

The US airline's apps tend to be very good, sending you real time notifications of gate changes and flight delays. On Delta, the app even took me directly to a rebooking function when it notified me of a flight delay during a layover in ATL. Very good UX.

Depends on the airline. United's app is terrible about updating on gate changes.

Jetblue is incredibly sensitive about any change and will do a push notification for it. (Your flight landed 2 minutes late, etc)

United's hit-or-miss: sometimes they're Johnny on the Spot

Other times? Yeah...not so much

(they're better when you enable push notifications for your flights (and I set up SMS notifications, too))

I don't fly too often, but when I do i've always found Google's flight info notifications to be quick and reliable.

I get Tripit Pro through work and I find that I usually get push notifications through the app before the flight information displays in the airport (!) have updated. YMMV.

I think it's evolving. For me it definitely used to be that way. I am a semi frequent Delta flyer and in the last year or so their app has become so reliable and useful that the only reason I need the airport monitors is that it is often a second or two faster to just look up at a big display rather that going through the clicks and swipes to get to the data in my hand.

A big improvement has been wifi availability and stability on the plane. Not so much because I can browse the web in flight, but because regardless of whether I get in flight wifi, the airline's app uses it for free. So now that wifi is reliably always there, I know what's up and where I'm going before I walk off the plane.

The apps are touch-and-go. Some are fairly reliable (assuming wherever you are you have decent data connection), others are abyssmal, and none of them seem to be consistent in either regard.

I have found, however, if you sign up for SMS updates through most airlines they appear to be quite reliable, and sometimes beat the in-airport departure boards by several minutes, particularly for gate changes.

I had a great experience with Lufthansa recently. Wasn't using their app, but received very punctual emails with delay and gate information, to the extent that the gate change email came at pretty much the same time as the announcement. Had a new flight, ticket, and gate automatically within about 10 minutes of missing my connecting flight as well.

I built and maintain one such service. Hopefully we don't have delayed gate information, but we DO ultimately rely on airlines, and we sometimes get very confusing information.

Alaska Airlines app works really well in my experience. Sometimes too well, as the gate agent starts the boarding process and suddenly a crowd forms at the gate before they have even announced anything.

The easyjet one is great, at least in CDG. Ryanair's is complete rubbish

Yea, I don't know why this reporter took such a negative tack. They deserve praise for continuing to deliver service during downtime.

This was a failure of a fibre connection to the internet and a cloud service.

Oddly enough, Gatwick's CIO was last week [1] telling journalists that he didn't like to rely on the cloud:

"The airport has used multi-tenanted public cloud in the past, but for that to work perfectly you need your telecom providers to work well, the path across internet to work, the hosting firm's telecoms to work, and their data centre support and monitoring not to drop. We have had examples where even in the public cloud using dedicated slices, you can suffer from noisy neighbours, and we just can't have that. Our operations are too important to us, so we need to keep it close. We use cloud for resilience, and we're very pro-cloud, but not for core services."

So it seems odd that departure boards were being fed from a cloud service. Maybe they hadn't been considered as a core service until now.

On the bright side at least the failure was limited to just the boards. The website, apps and check in desks all had the correct information.

[1] https://www.computing.co.uk/ctg/news/3034002/too-many-compan...

reminds me of the waiting room at the auto repair shop I work at. As an engine mechanic, its vital to make sure customers have access to up-to-the-minute stats on their minivan (or so im told by my boss.) Eventually someone knocked a stepladder into the screen and broke it, so we switched to whiteboard and engine techs had to scribble in status updates in front of customers like some sort of weird lecture.

Techs would mark down "NG" for no go, OK for ok, and HOLD...these all make sense to us, but not to customers. Even worse, once an hour or two a tired tech would shuffle into the waiting room study the board for a minute, and slowly cross out the name of a person. The whiteboard worked, but that day our shop was indistinguishable from a mafia den.

I struggle a little to see how there is a single point of failure here (a fibre optic cable). I’d have thought the screens were powered by a device plugged into a LAN. Would be interested if anyone knows the architecture of such screens and why there is such a single point of failure in the whole airport.

These screens build information from multiple sources inc. airlines, ground control, and in some cases airport management. They're typically run off-site at a data center since that provides high resiliency and some data is sourced from outside the airport anyway.

If the airport has a dedicated fiber channel between their data center and airport itself and it was damaged, this could be the result. The question I'd ask is why they have no fallback strategies (e.g. cellular, VPN over a less secure internet channel, etc).

This might be an area where they can explore graceful degradation, meaning that gate information would continue to display (from airport ground control), but some flight times might be missing.

It seems so strange that they wouldn’t have a redundant link (like the various options you suggested) or at least fall back to easily available public data (flight information sites) that would at least help some.

Perhaps they could have gone with a connectivity vendor with an extensive 4G network for backup if the (inexplicably) non-redundant fixed line was cut.

They're with Vodafone who has that.

Indeed. How embarrassing!

The problem is that it's the UK and mobile coverage isn't exactly world class here. You can get more consistent and reliable coverage towards the Russian border than you can around London.

I used to work for a major digital signage company and we had a huge contract with Verizon for cellular fallback at every site.

Some years ago, we lost access to our data center although it had triple network redundancy. What happened was that three cables were leaving the DC leased by three different companies. But a mile down the road, all three different cable entered the same physical pipe. When the physical pipe was severed, all three network cables went dark. We should wait until they announced details of what happened before we decide that we know what happened and offer solution ("If they only had redundant links")

If my memory serves, one requirement of high-tier datacenters is that they have redundant electric lines leaving the building from opposite sides and in opposite directions. I guess it must be the same for optic fibers.

I fully agree with you on your last sentence. One should wait for a bit more than a vague declaration reported in a newspaper before giving unsolicited advice.

Yes, they leave different ends/sides of the building, but you don't have much control on where they have utility right of way and whether or not at some point all of your lines share the same right of way (which is likely).

> high-tier datacenters

How many people (even tech ops profesionals), these days, know how to tell the difference? Of those that do, how many don't just keep their mouth shut because their bosses don't want to hear it?

These details are just trusted to "the cloud", instead.

Of course, the airport itself isn't a datacenter of any tier, and that "last mile" can be the biggest redundancy challenge, anyway.

I have heard this story before, with a twist that initially it was checked for, but the telcos changed it back some years down the road. I wonder what's a good way to audit this.

Or even the option of inputting the data manually on a local system, so they could use the big displays rather than some whiteboards!

As someone above already mentioned — the whiteboards themselves also convey the message of “something is wrong, you may want to verify this info” very well.

The medium is the message, etc...

While I agree that it's odd that there was no fallback, the whiteboards do immediately convey 'stuff is broken, you should check this board' pretty decently, as they would stand out like a sore thumb in the airport normally.

If there was a digital fallback, it would absolutely have to have something to tell commuters that the information might be stale (as someone frantically types up flight updates in a back room somewhere)

> why they have no fallback strategies

They do. It's the whiteboards. Whiteboards are cheap, resilient and robust. Exactly what one wants for a backup.

Well you also need to have people run around and update them.

Whats wrong with that? It seems to have worked. Sure not ideal, but better than canceling flights and shutting down the airport.

Obviously it’s better than shutting down the airport. But is it better than the alternatives Someone1234 proposed? That was the question.

I don't have the data of course but it does seem better. Those alternative are pretty expensive to build and maintain, have to be tested regularly and are prone to failure. For a situation that has a very low probability of happening, whiteboards do seem more reasonable, that's a very lean approach!

The chance of a single fiber line getting cut is not "very low", it's something that a big entity should expect to happen. Much like a power outage; I'm sure they have very good plans for that case. An extra modem is cheap as dirt.

Not saying there was anything wrong with it. The people who came in during their off hours to update whiteboards for however many hours are the real heroes of this operation.

That said it was probably very expensive :P

Theres a lot of fawning over the humble whiteboard here, just wanted to emphasize that they were part of a larger scheme

It's a good overall plan C.

But why does the data link itself have no fallback? An important data link should have multiple dissimilar backups.

> why does the data link itself have no fallback?

It's Gatwick, not Heathrow. Whiteboards are good enough. If every airport the size of Gatwick (and busier) implemented digital fallbacks, I wouldn't be surprised if the fallbacks took out the primary system more often than external causes. (I can’t remember the last time I used those screens over my phone. In a certain sense, the screens are now a backup.)

More precisely, if you consider the number of situations which (a) take down the primary system but (b) don't take out the airport, and try to cover those as cheaply and robustly as possible, for all but the world's busiest airports you come up with whiteboards. Localised power failure? Whiteboards. Surge? Whiteboards. Limited fire or flooding? Whiteboards. Fibre-optic cable gets nicked? Whiteboards or alternative digital system. Now compare the costs and risks of the digital system. Whiteboards are the work of a smart operator who probably saved their airport years of technical debt.

Gatwick is still a major airport, the 8th biggest in Europe or the 33rd at in the world.

The evidence is that whiteboards are ok, but with their location a second fibre link would not be difficult.

It sounds good in theory, but the practice is different.

“High availability” database measures are often less reliable than the normal, simpler approach.

Automated network failover equipment often causes more downtime than having multiple links and only failing over manually, or even depending on a single link.

Complexity is the enemy of robustness.

System failure is not just one simple category with known behaviours; different types of failures affect performance in different ways, and it’s hard to support all of them well. Just as exception-handling code is commonly the buggiest code out there, high-availability measures are commonly some of the most fragile parts out there, unintentionally causing more problems than they solve. They can be made robust, but it takes a lot of effort to do so, and frankly it’s seldom justifiable. Typically, simplicity is good. Definitely it’s easier to debug.

> Automated network failover equipment often causes more downtime than having multiple links and only failing over manually, or even depending on a single link.

Failing over manually sounds fine. If they can't swap over at all then that's a sign of fragility which will cause its own problems.

The data on the screens themselves could be represented by nothing more than a simple CSV file. A dial-up modem could handle the bandwidth to keep that up to date.

Or maybe the backup was in the same conduit and they both suffered backhoe fade - diverse routing is your friend.

According to this they have just finished rebuilding their network...

https://www.computerweekly.com/news/252441044/Gatwick-Airpor...

Oh I love it

[Gatwick has launched an app] for internal staff at the airport, such as shop assistants, baggage handlers and train station employees. ... He says the need for the app stems from some passengers entering “panic mode” and expecting to get flight information from anyone who appears to work at the airport. They may approach an employee in a non-passenger-facing department who has no more information than they do. ... “Passengers sometimes get more information from Twitter and Facebook feeds than from staff on the ground,” says Chacko.

“This is an attempt to bridge that communication gap and make everyone look smarter and more up to date.”

I wonder how well it worked today

https://www.computerweekly.com/news/4500269843/Gatwicks-Airp...

As someone who used to work in networking. Single points of failure are a way of life...

As is common in cases like this, there probably was a redundant link, but the link and its failover procedure probably wasn't recently tested.

Instead of redundant links, couldnt they have an option to update data manually?

Distantly related but when I was in Europe decades ago I was at a train station where they had a massive split-flap display ( I want to say at a couple places ). I loved watching it update.

Aesthetically it was so much better than big super bright screens.

What you're looking for is called a Solari Board, there's some mesmerising videos of them around, and a few places still have them!

I need one for my house .... maybe put the weather on it and have a button to cycle it.

You're in luck! https://hackaday.com/2018/08/18/custom-split-flap-display-is...

The main international terminal at Auckland airport still had a huge Solari board until just a few years ago (~2012).

Sadly, they had to go digital in the end due to spare parts becoming difficult to source.

According to Wikipedia, there are still some airports (eg Frankfurt) and many rail stations that still have them in active service.

The San Francisco Ferry Building has one that's in use today.

There's a company in the US that will make you one/rent you one, you can even control it via API. They sound really cool too.. I think I need to expense one for my office!

https://www.oatfoundry.com/split-flap/

Yeah if you have to contact them to get a price.... dang.

We can build a Split Flap for you!

They still have it at the Gare de Lyon in Paris

And Gare du Nord

South Station in Boston had one of these until maybe 5 years ago; sadly it's been replaced with a digital one. Though they do have a loudspeaker play a recording of the flapping sound whenever the display changes to alert travelers.

The board in Utrecht Centraal (the Netherlands) was taken down a few years ago, but you can still enjoy it in all its clackety goodness as https://itunes.apple.com/nl/app/blauwe-bord/id474162209 -- developed by the railroad company itself and showing real data off their open API.

Frankfurt Airport still has an enormous one (landside, not for those already waiting to board). It's absolutely beautiful to watch.

Yeh, whiteboard works but another more technical solution could be roll out a TV stand and hook up the TV to a laptop via HDMI and update schedule live through a shared doc (Google docs?). I mean, whatever works right...

That requires them to have a surplus of portable TVs and laptops, which I don't see why they'd have (and would be much more expensive to buy as backup than the white boards)

Also, the full-manual solution of having whiteboards on standby covers a much broader spectrum of failures.

As well as a whiteboard on standby, you also need 1 person per whiteboard to keep them updated. In an airport as big as gatwick, you probably can't get away with anything less than ~10 whiteboards (5 per terminal). Which means 10 people.

Not many companies have 10 spare people on site ready to act as emergency whiteboardmen.

Can’t one person update multiple whiteboards?

There are 21000 people employed at Gatwick airport. There are going to be a lot more than ten people on standby at any given time, waiting to take on exactly the kinds of surprise situations which are business as usual in a busy airport.

If your displays are openable and have the right connector you could just use them. However, as someone else pointed out whiteboards work in a lot more situations.

Why must the solution be technical?

That would require power sockets to not be a monumentally scarce resource inside an airport. Nobody is going to look for the TV rolled between a row of seats (that cause a dozen more people to have to stand or sit on the floor), facing a completely useless direction, to get flight updates. The airport isn't going to drag extension cords across the entire lounge without risking breaking somebody's neck.

They'll stare at the blank screens and wonder what's happening. A whiteboard in front of those screens, and other hot-spots, is far more considerate of how people behave inside a departure lounge.

Low-tech solutions to high-tech problems can be quite beautiful when executed well. It's a sign of good contingency planning because they anticipated no network availability, rather than betting on network redundancy.

I hope the few people who missed their flights were compensated in some way (refunds, seats on the next available flight...).

I believe you are totally missing the point. The displays were working properly, the internet connection was not.

If only London airports could print the gates on the tickets. But I guess they would miss out on that sweet revenue from consumption.

I think gates change when planes need to be re-routed.

That’s the exception and not the rule and when it changes it’s typically just around the corner and you can do show infos on the old gate.

I've found that when the gate is printed on the ticket, it is rarely correct. Even more so for connecting flights. So in the end, I still have to look at screens until I am actually at the gate.

I've found this too. The terminal usually stays the same because they are contracted to specific airlines, but the gate is fluid. For example, my wife is flying home today and her arrival gate was only announced a couple of hours ago (she's taking off just about now).

I think it depends on the airport & airline. Personally, in the US, I've had flights change terminals, not just gates. Terminal changes can be at least 20 mins on foot, even if you're fit and healthy.

Totally agree. In my experience, however, terminal changes occasionally happen but are unusual. Then, of course, there are places like Logan (BOS) where Terminal B has some sort of remote outpost- so even staying in the same terminal but changing gates can be a long haul to the connector because some of the gates are so far away

http://boston-bos.worldairportguides.com/terminal-maps.php

I agree and London is far from alone is this behavior.

There it annoys me the most. I always know which section of the airport my flight connection is and it’s a 20 minute walk. I rather go there now and sit down with my laptop and work undisturbed.

At high-volume airports gates aren't assigned until the plane has arrived from it's inbound flight which is often <1hr from departure time.

Checkin generally opens 24 hours prior to departure (some airlines now allow >7 days prior) - hence for the vast majority of passengers the gate is unavailable as ticket issuance time.

Most airports and airlines let you get gate information fro their websites/apps as well though..

I have a flight i take somewhat regularly at Heathrow. I think I only once did not find it at A20 and then it was around the corner.

So... massive IT failure and everything carried on working because staff was really competent? Damn the Brits are hung up on complaining

Went volunteering at a speed skating event once. One task I was asked to help with was mark the order of the skaters at the end of the race. I asked why that is and they explained that if all systems fail (1. Device on the skater's ankle, 2. cameras recording the race), then this ordering we note is what they will fall back on.

It seems like a lot of work people do is individual and idle when other systems take responsibility of many issues in a process. I'm glad that people worked closer together as a result of this failure!

I'm glad people worked closer together here, not that there was a failure!

After Deutsche Bahn was hit by wannacry Last year they too uses blackbords and announcers like in the old days. It was beautiful to watch.

https://www.telegraph.co.uk/news/2017/05/13/cyber-attack-hit...

Great example of "Improvise. Adapt. Overcome." in a bad situation - whether they had SOPs for this situation or not.

If I worked at Gatwick Airport, all my practice at whiteboards would finally be relevant to my job.

Hmmm... The flight boards for most of the airports around me here in Australia are available via the internet, on a public URL. [0]

Wonder why someone didn't just prop up PC's with large monitors or projectors around the airport, hooked into the WiFi to broadcast this info?

[0] - https://www.darwinairport.com.au/departures

Imagine having to update the whiteboard in real time though? Must be a right PITA, unless there is minimal changes going on with the flights (lack of delays or gate changes).

Talk about mvp :)

Simplest solution to the problem, love it

Whiteboard interviews vindicated. /s

Yes, even as ops grind to a dead halt, the company can still screen candidates looking for reasons/excuses not to hire them.

My first thought was that the whiteboards themselves failed, that this was some kind of The Onion article, but it isn't.

It's simply that the backup plan to the failing digital boards, are plain whiteboards.

Same. That is a really poorly worded headline.

https://en.wikipedia.org/wiki/Garden-path_sentence

This is the opposite of a garden path sentence: the first verb really is part of the sentence's main clause - i.e. the meaning of the sentence is "Whiteboards [are] used..." instead of "Whiteboards...fail".

A preposition swap (s/as/after/) should do the trick.

Have people forgotten they can use mobile apps?

Not everyone has a phone. Not everyone has that app (whatever it is... the airport, the airline, the travel agency), using an app is always harder than looking at the real-time displays especially if your hands are full of luggage.

And on top of that, not everyone's phone will be charged, and not everyone will have cell service (if traveling internationally).

The mobile apps are great when they work, but still need to be considered a convenience rather than a primary information source.

>Not everyone has a phone.

Corollary:

A few do have a phone, but it is not a smartphone.

The airport's whiteboard backup plan is interesting but my first thought was also that all of this info is available online or through the airline apps.

In my experience, changes often show up in Kayak ahead of the airport dashboards. I check apps or notifications before the official dashboards now.

According to the Times article flight information was not available online either.

Whiteboards are remarkable.

Reply on news.ycombinator.com