Hacker News Reader

Comment by planetjones | Hacker News Reader

I struggle a little to see how there is a single point of failure here (a fibre optic cable). I’d have thought the screens were powered by a device plugged into a LAN. Would be interested if anyone knows the architecture of such screens and why there is such a single point of failure in the whole airport.

View on HN

These screens build information from multiple sources inc. airlines, ground control, and in some cases airport management. They're typically run off-site at a data center since that provides high resiliency and some data is sourced from outside the airport anyway.

If the airport has a dedicated fiber channel between their data center and airport itself and it was damaged, this could be the result. The question I'd ask is why they have no fallback strategies (e.g. cellular, VPN over a less secure internet channel, etc).

This might be an area where they can explore graceful degradation, meaning that gate information would continue to display (from airport ground control), but some flight times might be missing.

It seems so strange that they wouldn’t have a redundant link (like the various options you suggested) or at least fall back to easily available public data (flight information sites) that would at least help some.

Perhaps they could have gone with a connectivity vendor with an extensive 4G network for backup if the (inexplicably) non-redundant fixed line was cut.

They're with Vodafone who has that.

Indeed. How embarrassing!

The problem is that it's the UK and mobile coverage isn't exactly world class here. You can get more consistent and reliable coverage towards the Russian border than you can around London.

I used to work for a major digital signage company and we had a huge contract with Verizon for cellular fallback at every site.

Some years ago, we lost access to our data center although it had triple network redundancy. What happened was that three cables were leaving the DC leased by three different companies. But a mile down the road, all three different cable entered the same physical pipe. When the physical pipe was severed, all three network cables went dark. We should wait until they announced details of what happened before we decide that we know what happened and offer solution ("If they only had redundant links")

If my memory serves, one requirement of high-tier datacenters is that they have redundant electric lines leaving the building from opposite sides and in opposite directions. I guess it must be the same for optic fibers.

I fully agree with you on your last sentence. One should wait for a bit more than a vague declaration reported in a newspaper before giving unsolicited advice.

Yes, they leave different ends/sides of the building, but you don't have much control on where they have utility right of way and whether or not at some point all of your lines share the same right of way (which is likely).

> high-tier datacenters

How many people (even tech ops profesionals), these days, know how to tell the difference? Of those that do, how many don't just keep their mouth shut because their bosses don't want to hear it?

These details are just trusted to "the cloud", instead.

Of course, the airport itself isn't a datacenter of any tier, and that "last mile" can be the biggest redundancy challenge, anyway.

I have heard this story before, with a twist that initially it was checked for, but the telcos changed it back some years down the road. I wonder what's a good way to audit this.

Or even the option of inputting the data manually on a local system, so they could use the big displays rather than some whiteboards!

As someone above already mentioned — the whiteboards themselves also convey the message of “something is wrong, you may want to verify this info” very well.

The medium is the message, etc...

While I agree that it's odd that there was no fallback, the whiteboards do immediately convey 'stuff is broken, you should check this board' pretty decently, as they would stand out like a sore thumb in the airport normally.

If there was a digital fallback, it would absolutely have to have something to tell commuters that the information might be stale (as someone frantically types up flight updates in a back room somewhere)

> why they have no fallback strategies

They do. It's the whiteboards. Whiteboards are cheap, resilient and robust. Exactly what one wants for a backup.

Well you also need to have people run around and update them.

Whats wrong with that? It seems to have worked. Sure not ideal, but better than canceling flights and shutting down the airport.

Obviously it’s better than shutting down the airport. But is it better than the alternatives Someone1234 proposed? That was the question.

I don't have the data of course but it does seem better. Those alternative are pretty expensive to build and maintain, have to be tested regularly and are prone to failure. For a situation that has a very low probability of happening, whiteboards do seem more reasonable, that's a very lean approach!

The chance of a single fiber line getting cut is not "very low", it's something that a big entity should expect to happen. Much like a power outage; I'm sure they have very good plans for that case. An extra modem is cheap as dirt.

Not saying there was anything wrong with it. The people who came in during their off hours to update whiteboards for however many hours are the real heroes of this operation.

That said it was probably very expensive :P

Theres a lot of fawning over the humble whiteboard here, just wanted to emphasize that they were part of a larger scheme

It's a good overall plan C.

But why does the data link itself have no fallback? An important data link should have multiple dissimilar backups.

> why does the data link itself have no fallback?

It's Gatwick, not Heathrow. Whiteboards are good enough. If every airport the size of Gatwick (and busier) implemented digital fallbacks, I wouldn't be surprised if the fallbacks took out the primary system more often than external causes. (I can’t remember the last time I used those screens over my phone. In a certain sense, the screens are now a backup.)

More precisely, if you consider the number of situations which (a) take down the primary system but (b) don't take out the airport, and try to cover those as cheaply and robustly as possible, for all but the world's busiest airports you come up with whiteboards. Localised power failure? Whiteboards. Surge? Whiteboards. Limited fire or flooding? Whiteboards. Fibre-optic cable gets nicked? Whiteboards or alternative digital system. Now compare the costs and risks of the digital system. Whiteboards are the work of a smart operator who probably saved their airport years of technical debt.

Gatwick is still a major airport, the 8th biggest in Europe or the 33rd at in the world.

The evidence is that whiteboards are ok, but with their location a second fibre link would not be difficult.

It sounds good in theory, but the practice is different.

“High availability” database measures are often less reliable than the normal, simpler approach.

Automated network failover equipment often causes more downtime than having multiple links and only failing over manually, or even depending on a single link.

Complexity is the enemy of robustness.

System failure is not just one simple category with known behaviours; different types of failures affect performance in different ways, and it’s hard to support all of them well. Just as exception-handling code is commonly the buggiest code out there, high-availability measures are commonly some of the most fragile parts out there, unintentionally causing more problems than they solve. They can be made robust, but it takes a lot of effort to do so, and frankly it’s seldom justifiable. Typically, simplicity is good. Definitely it’s easier to debug.

> Automated network failover equipment often causes more downtime than having multiple links and only failing over manually, or even depending on a single link.

Failing over manually sounds fine. If they can't swap over at all then that's a sign of fragility which will cause its own problems.

The data on the screens themselves could be represented by nothing more than a simple CSV file. A dial-up modem could handle the bandwidth to keep that up to date.

Or maybe the backup was in the same conduit and they both suffered backhoe fade - diverse routing is your friend.

According to this they have just finished rebuilding their network...

https://www.computerweekly.com/news/252441044/Gatwick-Airpor...

Oh I love it

[Gatwick has launched an app] for internal staff at the airport, such as shop assistants, baggage handlers and train station employees. ... He says the need for the app stems from some passengers entering “panic mode” and expecting to get flight information from anyone who appears to work at the airport. They may approach an employee in a non-passenger-facing department who has no more information than they do. ... “Passengers sometimes get more information from Twitter and Facebook feeds than from staff on the ground,” says Chacko.

“This is an attempt to bridge that communication gap and make everyone look smarter and more up to date.”

I wonder how well it worked today

https://www.computerweekly.com/news/4500269843/Gatwicks-Airp...

As is common in cases like this, there probably was a redundant link, but the link and its failover procedure probably wasn't recently tested.

As someone who used to work in networking. Single points of failure are a way of life...

Instead of redundant links, couldnt they have an option to update data manually?

Reply on news.ycombinator.com