Hacker News Reader

Comment by DiabloD3 | Hacker News Reader

I mean, ultimately, thats how Google routes internally.

IPSec-equivalent, VXLAN-equivalent, IPSec-equivalent.

Prevents any compromised layer from knowing too much about the traffic.

What gave you that idea? Internally, Google uses GRE/GENEVE-like stuff but for reasons that have nothing to do with "preventing compromise" or whatever, but because they're carrying metadata (traces, latency budgets, billing ids.) That is to say, encapsulation is just transport. It's pretty much L3 semantics all the way down... In fact, this is more or less the point: L2 is intractable at scale, as broadcast/multicast doesn't work. However, it's hard to find comparisons to anything you're familiar with at Google scale. They have a myriad of proprietary solutions and custom protocols for routing, even though it's all L3 semantics. To learn more:

Andromeda https://research.google/pubs/andromeda-performance-isolation...

Orion https://research.google/pubs/orion-googles-software-defined-...

What gave me that idea? Talks and research papers from Google network engineers over the past decade.

Where are you getting at VXLAN-equiv, IPsec-equiv, etc. specifically? ALTS/PSP is not "IPsec-equivalent"

The last time I was there, there were many layers of encap, including MPLS, GRE, PSP, with very tightly managed MTU. Traffic engineering was mostly SDN-managed L3, but holy hell was it complex. Considering that Google (at the time) carried more traffic than the rest of the Internet combined, maybe it was worth it.

Internal is fine because you control things like MTU so you don't have to worry about packet fragmentation/partial loss.

That seems like an awful amount of overhead for questionable gain.

Links between, and in between data centers use so called jumbo frames with an mtu of over 9000. Not joking.

Worth mentioning that links at home can use them too, jumbo frame support was rare at one point but now you can get them on really cheap basic switches if you're looking for it. Even incredibly cheap $30 (literally, that's what a 5 port UniFi flex mini lists for direct) switches support them now. Not just an exotic thing for data centers anymore, and it can cut down on overhead within a LAN particularly as you get into 10/25/40/100 Gbps stuff to your own NAS/SAN or whatever.

Reply on news.ycombinator.com