Hacker News Reader

Comment by ctoth | Hacker News Reader

> To save money, the FAA elected not to build a new STARS server in Philadelphia to support the move. A new server alone would require tens of millions of dollars, as well as installation of new internet and power infrastructure.

> Instead, it elected to send a “mirror feed” of telemetry from the STARS servers at N90, traveling over 130 miles of commercial copper telecom lines, with fiber optics to follow by 2030.

> The annoyances of traditional cable internet — frequent lag, dropped sessions — are probably familiar to those who stream video or play games online. But for air traffic controllers, even the smallest service disruptions can become dangerous.

So LOL what, they just ... piped it over the Internet? Also can someone make sense of this "new server" costing millions of dollars? Presumably it's not the cost of a server, which is orders and orders of magnitude less than that?

View on HN

Aerospace is hard.

An ATCS like STARS needs to feed from multiple different OT and IT sources like radars, weather stations, other TRACONs, etc and is implemented in it's own airgapped environment.

It can get very pricy very quick. On top of that, the FAA's budget has been sclerotic for decades now after the 1980s era union action and the 1990s era national cost cutting.

And finally, it is a political organization, and NATCA is a fairly prominent union within the AFL-CIO, and could make the lives of NJ representatives hell for pushing reassignment out of Newark.

> Aerospace is hard.

I would believe it was hard. And maybe it still is if you’re unwilling or unable to take advantage of modern technology.

Current low-cost equipment can easily send 10 or 100Gbps over long distance fiber links. Depending on how quickly you want to fail over when a link or an entire switch, router or rack fails, there are plenty of options that make various tradeoffs between failover latency and bandwidth, all the way up to completely duplicating all the traffic on redundant routes. I would bet that the entire aggregate traffic needed for air traffic control in a region is well under 10Gbps. And 10Gbps dedicated links or leases or (effective) purchases of dark fiber are not expensive on the scale of the FAA. Air traffic should use a network with a lot of redundancy, so maybe multiple those low costs by something like 5.

Heh, have fun hooking legacy systems to high speed networks without significant testing.

If seen plenty of old stuff crash because you'll have some ancient serial device with a limited buffer and someone jams a faster link in-between. All of a sudden you have a much larger amount of bandwidth delay product and the system doesn't handle a few megabytes of data getting lost on the line when it bursts for some unexpected reason. On the old fixed line that just couldn't happen.

I doubt that 130 miles of copper is the public Internet. It’s presumably some legacy telecom system that depends on a bunch of generally fairly well made but thoroughly obsolete hardware.

Keep in mind that Ethernet over copper is only specified to ~100 meters. Long distance copper networks have been obsolete for a few decades.

Here in my EU state, air traffic control still used leased copper twisted pair for server link over ~150km as of at least 2015 or so. It's pretty reliable and not "public internet" at all - just two modems with known performance and known latency taking to each other.

Might be something like that in this case as well.

Sure, this kind of technology works fine. Except that it might be difficult to find people who how maintain or repair the fancy (pressurized? oil-filled?) copper cables or the modems, and finding parts might be interesting.

I did some enterprise network work ~20 years ago, including fiddling with some inter-building links, rummaging through closets, and visiting the inside of a legacy campus-scale analog phone exchange, and I’ve never even seen the kind of equipment that can send data at an appreciable speed (even by 1990s standards, and even with repeaters) over copper at a range like 100km.

In contrast, single-mode fiber has improved over time, but it’s not obsolete, and it has maintained a remarkable degree of compatibility over the years. New transceivers largely work on old fiber, old transceivers work on new fiber, etc.

Those are all good points. I do wonder if - as you mention - it might just be leased fiber.

I mean its a Verge article.

They dont know the difference between copper and fibre.

And yea fibre was in the 80s too. No reason for new deployments to be copper.

It’s called a dry loop. It might be real with repeaters at the various COs, or it might be partially virtualized and sent over a fiber trunk. Usually somebody leading a dry loop does so for the relatively extreme reliability, even though the speed is not very high.

Why not pipe it over the internet? A few redundant ISPs at the endpoints should be as reliable as a private run of fiber or T1 lines or whatever bespoke solution they think they need?

Doing so sticks billions of transistors between A and B, all of which have to work perfectly billions of times a second. If you just run fiber, you need to put an amplifier in every 50km or so, other than that it’s all just buried glass.

The fiber is already there, and it gives you guaranteed capacity, even during a major power outage. And it guarantees no one is changing out a switch or a router at a bad time, etc.

It’s just a lot less to control and to go wrong.

As soon as it's on the general Internet, any ill-meaning enemy can just force the VPN endpoints offline by blasting them with junk traffic.

That can be mitigated effectively in this case using an IP whitelist. All unrecognized traffic can simply be dropped.

...Until they switch to a bandwidth exhaustion attack.

True, saturating all routes to a host is quite feasible, but with sufficient redundancy (say with multiple PTP wireless links, which can be done quite cheaply) you'd have to knock the Internet out in such a huge area we'd be talking about WW3.

There's no argument that a private line is ideal for critical infrastructure, but if they must make do, there are ways to make it work.

People managed to shoot Wikipedia off the Internet as recently as 2019.

Anyone not behind one of the major CDNs can be targeted and taken down. And bandwidth-exhausting a few routers in front of the target is certainly possible, it's just not profitable for cybercriminals - but for a nation-state? No problem at all to muster a few terabit/s of capacity for them.

Nah, probably on leased lines like t1s/t3s. Airport telecom infra isnt always the best.

When they say "server" it means the (cheap) hardware and the (very expensive) software to drive these complex and critical systems.

If it's mainframe, and there is no reason to believe it's not given the strict requirements, the hardware is expensive as fuck and the people who can keep these beasts alive are just as expensive.

I think they are running on old Sun Ultra workstations, not mainframes. Hardware easily replaced and upgraded with much faster, cheaper options. Difficulty is much more on the software side.

> I think they are running on old Sun Ultra workstations

It looks like it, though in this brochure there is a bunch of what look like Sun rackmount servers in the background as well:

https://web.archive.org/web/20120930072126/http://www.raythe...

Very interesting. Looks like the LITE STAR runs on the Workstation and the full STAR deployment runs on the servers.

But can you even buy these now? A new STAR server setup must be x86 and virtualized,no? Maybe even cloud?

Agree, I didn't mean 10k servers, I mean 100k servers (eg); and I also meant $10M persons. Trying to (poorly) state a two orders spread.

A "Server" is just layman's speak for that big room with all the noise and blinky lights. Including all the hardware, networking and software that goes with it. Of course you're not going to run critical infra on a single server :)

Though I have to admit I have seen government operations where the "server" was an old dell optiplex desktop lying on its side in a broom closet without ventilation, a post-it with "IT SERVER DON'T TURN OFF", a spiderweb of cables running through the closet and the "server" fans screaming for air trying to keep everything cool in the enclosed space. I'm not kidding.

I mean, I know, government. Small local welfare-related org. Shoestring budget. Sure, that sucks. But at least you can make sure it's tidy and the cabling doesn't look like shit. Jeez. I didn't imagine I'd still see that in this century. Do people no longer take pride in their job? They hadn't even activated the "AC Power on" in the BIOS so after electrical maintenance they had to wait for the "engineer" to press the on button again.

There's no budget to fix whatever you messed up when redoing the cabling, which we also didn't really have the budget for. Everyone who knows what the box actually does has retired or died, so nobody wants you in the broom closet shifting dust motes near a working system we can't replace.

What are leased lines?

Old tech. You’d literally lease a line from ATT or whatever Bell company covers your area. A reserved private connection. Modem at each end. They ran at about 56Kb/s IIRC.

> pp. lag

> hold all traffic

> rdy?

> gogogo

Reply on news.ycombinator.com