Google's Subsea Fiber Optics | Hacker News Reader

Google's Subsea Fiber Optics

cloud.google.com

181 pointsshade2396 comments

Unrelated, but if interested in ocean cables check out “A Thread Across the Ocean” by Steele. It’s the story of the first transatlantic cable. It’s a riveting read that is hard to put down. Full of interesting technical details intertwined with the stories of the characters involved. Highly recommended!

Seconded! I haven't read through a book that quickly in quite a long time.

Also The Victorian Internet by Standage on the history of the telegraph:

* https://en.wikipedia.org/wiki/The_Victorian_Internet

One of my favorite books!

Some years ago I bought a dozen extra copies to give away. Just last weekend I was hiking with a few friends. One mentioned that I'd given him a copy, and he started telling me stories from it. Especially what a revolution it was when you could communicate across the world in an instant.

We got to talking about the history of undersea cables. I mentioned a very cool interactive map of the current fiber cables and promised I would look it up and send him a link.

So, before I forget, here it is:

https://www.submarinecablemap.com/

I think it's also obligatory to mention the longish magazine article "Mother Earth, Mother Board"

https://www.wired.com/1996/12/ffglass/amp

Richard Steenbergen has regularly given the presentation "Everything You Always Wanted to Know About Optical" at NANOG over the years; October 2019:

* https://www.youtube.com/watch?v=nKeZaNwPKPo

APNIC/NZNOG had a good presentation focusing on sub-sea optical stuff (January 2020)):

* https://blog.apnic.net/2020/02/12/at-the-bottom-of-the-sea-a...

For longer distances (>100km), you want to do a search for "coherent optics".

On my desk sits a paperweight. It is a segment of some ancient undersea cable. About 2 inches in diameter, sheathed in lead with eight main wires and numerous smaller ones in a circular arrangement. Weighs a ton. My dad worked for AT&T and received it as a gift...

omg... I need something like this!

I wonder how long it will be before we see the first hollow-core fiber subsea cables. They are 50% faster, and tests from the last year or two have seen record low signal losses.

https://www.laserfocusworld.com/fiber-optics/article/1419605...

https://www.ofsoptics.com/wp-content/uploads/Hollow-Core-Fib...

> They are 50% faster

No, pretty sure light still travels through them at C. What they can do is carry more data, largely by having lower error rates.

'c' is dependent on the medium. The value of c as it is commonly known, 300 million meters per second, is in vacuum. Light traveling through other media is affected by its index of refraction, in the case of silica fiber, that is approximately 1.5 so radiation propagates much slower through silica than it does a vacuum. Since gases have low refractive indicies already, within a hundred ppm or so of a vacuum, you could essentially round air to 1.

I've heard that high frequency traders are interested in Starlink's planned laser links because they could open up routes that are faster than traditional terrestrial fiber.

I think that if you are a HFT, you probably have a server set up next door.

The suggestion is about trading across multiple exchanges, for example between London and NY. Going via Starlink is potentially quicker than a fiber under the Atlantic.

They will have servers “next door” to the exchanges, but need the servers to have incredible low latency connections to each other.

Yeah, that would make sense. There are links that have been built by various HFT firms and banks [0] [1] that use microwaves instead of fiber buried in the ground simply because of this speed-of-light-in-media limitation. They can shave a few hundred nanoseconds (or something, I don't want to do the math right now) because of a higher signal propagation speed and get a trade in faster than their competitors. Same thing with a laser link like this.

Edit: cf.

[0] https://arstechnica.com/information-technology/2016/11/priva... [1] https://www.reuters.com/article/us-highfrequency-microwave/l...

I know the main people behind much of the hollow core work being done. Much of their financing is coming from HFT related firms.

Light only travels at C in a vacuum (so it doesn't actually travel at C in a standard fiber optic cable, it's actually much slower).

"c" is the speed of light in a vacuum. Traditional fiber optic cables are very much not a vacuum, with an index of refraction of ~1.5, so light travels through them at ~2/3c. In contrast, light actually travels at nearly c through hollow core cables.

It's so nice when someone other than myself is confidently incorrect.

I might be totally wrong; It seems likely to me that, due to capillary action, if a hollow undersea fiber gets physically cut then seawater would flow into the hollow center.

The ends of the fiber might be at different depths with a pressure difference that could move water a long way into the fiber. I imagine the length that water got into would be ruined even if the water was pushed out again.

I conjecture that undersea hollow-core might end up being expensive to maintain.

IIRC individual fibers are terminated every Nkm at a repeater. Not that it wouldn't be spendy, but I would also conjecture replacing a segment of fixed length instead of just gluing the ends back together might still be a reasonably strong constraint on unplanned repair cost (and also probably providing a pretty strong lower constraint as well--notably higher than solid core).

I don't think they are terminated every Nkm anymore. We have been able for quite some time to re-energize the signal directly in the optical core without needing to convert it to electric then back.

https://en.wikipedia.org/wiki/Laser_pumping

If a cable is long enough to require amplifiers, they are spliced in. Couplers/circulators need to be in line with the doped section of fiber that forms the laser amplifier, both before and after. One of the couplers injects the pumping beam from another laser into the doped section. This necessitates splicing at every amplifier. Also, almost every strand in the cable would require amplification.

In addition there are feedback, failover and monitoring functions that require more optical components to be included, and it's likely that this type of functionality will increase as demand for improved latency and reliability increases, and new cable networks are built.

If there's a leak that would allow water access to the core, the signal's already gone.

And, a hole that small in a block of glass could withstand a titanic amount of pressure.

Cables like this would have spare strands to be able to handle the situation in which a single strand was compromised by a break or pinhole, and possibly a different type of jacket on each strand to manage the stresses. If a section is badly damaged beyond the point of patchwork repair, then the only hope from that point is that the customers using the strands/wavelengths on the cable have an active alternate path while the whole section is dug up and replaced.

I may be even more wrong, but maybe this leads to a more modular way of building/laying these "cables".

I imagine that you need less of them, because "faster" also means longer range until attenuation kicks in. How much more range/less amplifiers compared to current state of the art I don't know. I'd think it should be possible to lay down a new part the whole length between two amplifiers/laser pumps/couplings/ and "plug" it in by remote controlled submersibles, instead of lifting up the broken ends, and "splicing" it on the ship. Like pre-made Cat-5/7 with RJ-45 plugs, so to speak :-)

With hallow-core, Spread Networks[1] could be back on top again!

1 - https://en.wikipedia.org/wiki/Spread_Networks

The "50% faster" stat is talking about latency, which is not really what subsea cables are about (massive bandwidth).

The more important figure is the attenuation at frequency bands used in telecommunication (C-band and L-band). According to the article, hollow core fibers are still much worse than solid core fibers (0.28 dB/km vs 0.14 dB/km).

While I was at AWS I managed to catch a couple of internal presentations from James Hamilton where he went deep into their global network infrastructure. It’s something that’s way outside my wheelhouse but that I find fascinating nonetheless.

In particular I always found the discussion about dealing with failure of global infrastructure that’s sitting in the ocean floor particularly intriguing. One non-obvious (to me at least) aspect was what do you do if there’s a failure of the power feed equipment that’s driving the cable. In most cases the answer was “double the voltage until you can repair it properly”. He explains it far better than I ever could on his blog: https://perspectives.mvdirona.com/2017/01/cs-responder-trans...

If you're interested in learning more about how a WSS (Wavelength Selective Switch) works under the hood, there are 2 main methods:

* An array of MEMS micro-mirrors which turn to reflect light, or

* Finisar's LCoS based method, which uses a 1" LCD TV display to display a gradient pattern which literally bends light.

It looks like the undersea cables use Finisar's (now II-IV's) WSS. I think the idea of bending light under software control is pretty cool.

Finisar's white paper showing optical path and other possible uses is here: https://www.amstechnologies.com/fileadmin/amsmedia/downloads...

https://web.archive.org/web/20150513001347/https://www.amste...

A bit old now, from 1996, but the classic book-length Wired article "Mother Earth Mother Board" by none other than Neal Stephenson is a peek into how this actually gets done from the physical cable laying pespective: https://www.wired.com/1996/12/ffglass/

came here for this

The internet includes many components: semiconductors/electronics/chips, hardware, fiber optics, communication systems, networking, wireless, software, etc. There is a lot of work that must be done in different parts of this stack for this system to work.

Yet, the private sector focuses mostly on the software part, or services. I have rarely seen a start up on improving optical fiber or electronic chips. The public sector builds the infrastructure, often following decades of investment and work. People working on infrastructure either work for the government for pennies or, if they haven’t yet lost their jobs to outsourcing to developing countries, have difficulty finding employment. The profit goes to consumer companies focused on software or services; worse, these companies claim credit for the whole Internet.

Obviously CapEx will be large for a company with a product on infrastructure; there are monopolies; customers will be large operators, etc. Still, are there resources to better understand this issue? It always seemed to me a scam.

Also, will the situation change for “hardware”startups/companies?

Tragedy of the commons, its the same reason there are big car companies but not big road companies.

How much of it is government lockout/red tape?

Road companies seem like a natural monopoly so that doesn’t really apply.

But there are big telecom companies and big chip companies, etc.

> The public sector builds the infrastructure, often following decades of investment and work.

It does? I don't think I follow. In your list of components, every single one of those is, at least in the US, largely or almost entirely handled by private companies.

The big semiconductor companies are private. I actually don't think there are any notable public entities that make their own chips. Hardware companies (I'm assuming you're talking about things like motherboards, routers, switches, etc) are private. Fiber optics/communications/networks are laid almost entirely by private telecom companies (and there's actually a big push to take this away from private companies and make ISPs be government entities). The article that you're commenting on is all about a private entity investing money into laying fiber and improving the protocols that communicate over it.

>I have rarely seen a start up on improving optical fiber or electronic chips.

There are a lot of SMBs working on chip design. I'm less familiar with fiber, but a quick google shows at least a couple, all private.

Oh no!

Consider high speed optical communication. The invention of transistor, laser, optical fiber, optical amplifiers, modulators, communication and information theory, DSP, and many other important technologies, was funded by taxpayers for a long time. You may say, some of it happened in old bell labs, but bell labs was a state-subsidized monopoly, rather an exception, and even things like information theory were really developed at MIT and other public research labs (Shannon had an office at Bell Labs for some time but had limited contact there).

Consider machine learning. From 1940s to 2010, neural networks were funded by governments grants, and developed at universities. The governments even funded building practical applications out of deep neural networks, for example, in the area of speech recognition, unsuccessfully. The periods are now called AI winters. It’s really in the past 15 years that the subject has become practical and companies are building products based on it. FANG is capturing the profits of this decades-long research and development.

Today there are millions of graduate students and postdocs and professors at universities performing risky research and experiments. A lot of it doesn’t pan out. The public bears the risk. Once an idea begins to work, it’s handed over to private sector for further development and bringing the work to market.

Most of the research papers published at conferences and journals, Nobel prizes, Fields Medalists etc come out of universities, not startups or FANG.

Yes, Google installed fiber cables (perhaps even bought it from Corning or the like, and also contracted out the installment). But that’s not infrastructure R&D.

I think there is no denying that basic long term research is done by and large in public sector. Private sector is still focused on short term product research.

> The public sector builds the infrastructure

It's a little off your point but there was an amusing book published a few years ago which went through twelve, from memory, technologies on which the iPhone was totally dependent and which had all been developed out of the public purse.

Can't remember what it was called so if anyone recognises it I'd be interested to hear.

There is this book by Mariana Mazzucato: The Entrepreneurial State: debunking public vs. private sector myths

I haven't read it, but from the TOC, it looks like these are the technologies that are "invented" by public sector that the iPhone (and other smartphones) rely on:

* Giant magnetoresistance (GMR), SPINTRONICS programme and hard disk drives

* Solid-state chemistry and silicon-based semiconductor devices

* From capacitive sensing to click-wheels

* From click-wheels to multi-touch screens

* Internet and HTTP/HTML

* GPS and SIRI

* Battery, display and other technologies

ref: https://cms.marianamazzucato.com/wp-content/uploads/2021/04/...

Summary HBR article: https://hbr.org/2013/03/taxpayers-helped-apple-but-app

The barrier to entry for a software company is nearly non-existent these days, where it's still really high for hardware. I can learn how to code, write an app, market and sell it all from a computer I can buy from Target or Walmart. Hardware is significantly harder to learn, then I'd have to source components, and scale beyond a prototype takes a significant amount of money. Scaling software is done with a button. You see more software made in the private sector because it's so much easier to do.

I'm a huge supporter of publicly funded research. I think the things that have come out of DARPA, NASA, the NSF and other US funding bodies are incredibly important.

Having said that, what you've said is basically wrong.

> I have rarely seen a start up on improving optical fiber or electronic chips.

Unless you are trying to be pedantic by saying it has to be a startup rather than a private company this is completely wrong.

TSMC, Intel, Samsung and IBM do most of the work improving electronic chips manufacturing processes.

I'm not very familiar with optical fiber work, but I know NTT in Japan does a lot of R&D in the area.

Right, those are examples, but what’s the proportion of hardware and software companies in US (big companies or startups)?

How much of VC funding goes to companies focused on services or software vs hardware or infrastructure?They actually write down their areas of focus on their websites. I don’t see many of them stating they fund companies developing better chips, optical amplifiers, fibers or other pieces of infrastructure. It’s mostly services, software, apps, consumer applications, and similar.

And to my point, how much do you make if you are a hardware engineer at Intel (a large hardware company), vs a software engineer at say Google (a large software company) with similar years of experience?

VCs don’t invest in chip manufacturing improvements because improvements are incremental so there's no return. But the point remains: it's private investment.

They do invest in new chips. There's a bunch of "AI" chips funded by VC which are focused on the outside returns available by improving training performance. These are all funded by private investment.

>Yet, the private sector focuses mostly on the software part, or services. I have rarely seen a start up on improving optical fiber or electronic chips. The public sector builds the infrastructure, often following decades of investment and work. People working on infrastructure either work for the government for pennies or, if they haven’t yet lost their jobs to outsourcing to developing countries, have difficulty finding employment.

Citation desperately needed. How you got "the government pays for our internet infrastructure" out of an article about Google paying for a new Subsea cable, I do not know.

I implore you to do a tracert right now to hackernews, and lookup who operates the ips of every router it hits in between. Chances are incredibly good not a single hop is on a government network. (true for the U.S at least)

A lot of the subsea fiber is laid down by and funded by private organizations. I’ve worked with a couple that intentionally avoid any public dollars to avoid the hassle and delays that come attached to that money.

I make 340Tbit/sec about 1.1x10^11 GiByte/month. GCP premium tier networking is priced at $0.08/GB, so at 80% load that cable would, very naively, have the potential to bring in $7B/month in revenue.

I'm sure they only take in a fraction of that, and their costs are substantial. But even so... cloud bandwidth is overpriced.

8c is for transit to outside of their network. For inter-region it’s like 1-15c depending on regions. 1-2c for us/europe which is probably overwhelming majority

Also 80% utilization seems just ridiculously high to me but maybe at goog volume it’s doable

I remember seeing a cloudflare post about AWS bandwidth pricing where they estimated something like 20% utilization? I don’t remember where though but I think they can approximate pretty well.

Google regularly runs these at 100%. According to the B4 paper:

"""These features allow many B4 links to run at near 100% utilization and all links to average 70% utilization over long time periods, corresponding to 2-3x efficiency improvements relative to standard practice"""

How they increase efficiency by using it at 100%? They would stop some lines at night?

Haven’t read the paper yet but guessing just start egressing from random pops on the way when the pipes are full

Well, you wouldn't want to hit 80% on Day 1, as you would have no room for growth. Perhaps 50% and after a few years† you'll hit 80% and start planning for a new cable.

† The video said this started five years ago, so there appears to be a lot of lead time that is needed.

Yup, but it will continue to be absurdly overpriced because the CapEx is massive and governments are totally okay with oligopoly.

You’re thinking about average throughput, while these cables need to be provisioned for max throughput, which can be completely different.

Having said that, cloud bandwidth is indeed overpriced; but at the same time, given that Google Cloud is still burning money, can it perhaps be argued that bandwidth is one of the money makers that allow for other services to be free?

I recall that from the old webhosting days, this was already a common tactic of the providers: lure people in with cheap servers, sometimes even at a loss, and earn money back with bandwidth.

Is it good for the economy though?

Resources are used depending on prices. If the costs for providing bandwidth are low and everything else is expensive, but the prices are the other way round, then the economy optimizes to waste resources. That's not sustainable.

Overpriced is an understatement. It's like 20x as expensive compared to things like colocation.

With the exception of high-storage/bandwidth websites like video hosting platforms, bandwidth scales linear relative to audience/reach so the high cost is a justifiable expense. We haven’t seen a race to the bottom with bandwidth like we have with storage because the usage of bandwidth implies the product is being used.

Furthermore, software (as a product/service) has the lowest marginal cost of nearly any product. Given the cost it takes to have one more customer on your platform is some nominally small amount of bandwidth (which depending on the product, can be sub 1 gigabyte per month) the additional expense is easily justified.

When they say a single cable can deliver 340 Tbps capacity, do they mean a single fiber strand, or a bundle of strands in a sheath that we know as "cables"?

Pretty sure they mean the sheath that contains the bundles of fiber cables.

Generally the throughput for a single mode fibre in the C + L Bands (the wavelength regions used for telecom applications), is about 100 Tbit/s for a one span link (50-100km) for a submarine cable across transatlantic distances IIRC the record is around 50-70 Tbit/s. This is research demonstrations, so the 340 Tbit/s would be for a cable with plenty redundancy. Also note that fibres are used in one direction only (one of the main reasons is that one would otherwise create a very long laser), so for duplex operation you need to double the amount of fibres.

It seems like the expense would be in the armored outer cable, repeaters, and labor for laying the cable but perhaps at those distances the glass cost matters? Still it seems like you'd want to cram as many fibers into the cable as possible. There must be some limiting factor that prevents you from putting 1000 strands or 10,000 strands in a single cable.

Adding more fibres also means more power needed for more repeaters.

Apropos HN and fibre cable laying, I'm told Makailay cable laying software rents out for $20K/Month. Could be more now - this is ~10/yo information.

There's lots of niche markets out there.

https://www.makai.com/cable-software/makailay/

> A message took over 17 hours to deliver, at 2 minutes and 5 seconds per letter by Morse code

A letter in Morse code is made of up to four “dits” or “dahs”. Why would it take more than two minutes to send one letter?

You can read a transcript of the first conversations on the transatlantic telegraph[1]. Basically: the signal was very weak, they needed a lot of time between symbols, and they needed to repeat a lot.

https://www.google.com/books/edition/Report_of_the_Joint_Com...

The signal was weak. An attempt at fixing the problem, boosting the voltage, caused the insulation to fail. Later cables added repeaters along the way to maintain the signal. https://en.wikipedia.org/wiki/Transatlantic_telegraph_cable#...

For those more interested in this topic TE Subcom (now just Subcom) has some cool videos about the process of deploying and repairing these submarine cables. Just search for te subcom on YouTube.

Did they say that this will power the next (Google) Wave™? ;)

Are we gonna get the Google Moon Cable anytime soon? :-D

Is that a real idea? Can’t Google right now unfortunately!

Not that I know of but I thought it might be funny!

So I'm not suggesting we don't lay more undersea cables, but has anyone done any studies on what damage laying these cables does to sealife/coral/habitats on the sea floor?

I love scuba diving and I've personally seen many coral reefs wrecked from boat anchors and fishing nets, do undersea cables do similar damage? (I'm guessing yes).

Funny how tech people (like me) care about pollution from emissions and plastic unless it’s related to rocket launches or huge plastic cables on the ocean floors. Then, nobody mentions it.

Do rocket launches or cables on the ocean actually contribute a significant percentage to world pollution? I would imagine it's insignificant.

Rocket launches definitely do.

Do you have any numbers on that?

Do you have numbers that it doesn’t pollute?

Benefits vs cost.

Rocket launches typically develop and support bleeding edge technology. And are relatively infrequent compared to regular travel flights.

The "huge" plastic cables are 7-8cm (~3 inches) diameter. So big, but not gas pipeline big. And because of THEIR existence many people are NOT taking those regular travel flights they otherwise might.

Not saying we can't do better. But people closer to it and smarter than me would know.

I don't understand why or how the people in the video are so glowingly happy/smiling. Is some point being made there?

People happy to share their work. Is that some kind of problem?

The video doesn't look like that, it looks more like everyone was told "you have to smile more!!"

This seems a super-cynical take. Some people are smiley and happy naturally. You might pick them to be in a video.

I was going to write that this was demonstrably untrue, but then I saw the muted-microphone shot of an interviewee laughing without context before cutting to a straight-faced interview segment that appeared more natural (at time = 80 s), which was quite possibly recorded after the straight-faced segment to make the video's happy tone consistent: https://youtu.be/N0ng8R0_Tis?t=80

Feeling comfortable at work usually doesn't involve grinning into an abyss like at one's best friend.

This looks forced and cringey.

It's an overproduced PR piece

What would be the correct amount of production?

candid unrehearsed interviews

Good luck getting anyone at a publicly listed company to sign off on a video promoting a $xxx million project with candid unrehearsed interviews.

A presentation at a conference like NANOG, APNIC, IETF, etc. See my other comment:

* https://news.ycombinator.com/item?id=31426614

Also crazy camera angles showing the backs / sides of people talking.

Have you never seen a video produced for or by a company before?

> Is some point being made there?

Yes! That everything is fine, everyone is happy and if you're not happy, then the only sane conclusion is that there's something wrong with you.

Reply on news.ycombinator.com