Hacker News Reader

Comment by crb | Hacker News Reader

Google Cloud Platform builds its own: https://www.wired.com/2015/06/google-reveals-secret-gear-con...

Google (corporate? maybe not Cloud) and Facebook still buy a lot of vendor gear, though it's mostly Arista. Google and FB have investments in running their own whitebox gear, but they use their own purchasing size as a carrot with companies like Arista. "If you can build this feature in 3 months, we'll buy X units. Otherwise we'll build it in house." Arista has much more resources to bring to the challenge, and they generally get the feature built quickly.

Sometimes I wonder if the "in house" networking projects are just a ruse to get vendors jumping whenever you call. The brainpower and talent that a company like Arista can dedicate to projects is much more than a tiny division inside Google or FB. Not hating on Google or FB, just trying to figure out their game.

Arista is also more flexible on licensing their software such that you can add your own patches after the fact.

It makes (business) sense to build your own gear when you operate at scale. Typical use cases are TORs and small table routers used for data center Clos topologies. Both often make use of merchant silicon which come with SDK's.

Still trying to understand the ToR case. They're pretty basic with enough ports, BGP, and mgmt agents to do just what you need. That said, these things are _stupid_ cheap now. The Quanta switches that AWS used/use couldn't have cost more than $200/each direct from ODM. Maybe it makes sense at scale if you only need 38 ports in a rack to build your own so that you save on power costs from cutting out the extra 16 ports or so? Improving PUE and opex might be the big driver here?

I mean, these ToR switches really are super cheap if you buy direct from ODM. For designing your own ToR, you'll need at least a team of 5-6 people each making $400-500k annually at a company like Google or FB. How much is the cost savings per ToR switch in capex? $10/switch? In opex? I think it might make opex sense.

You'll build and run your own mgmt plane OS for sure, but that's tangential to building your own hardware.

Right - the hardware is already built. PHY, chipset, host CPU/disk/RAM, etc. It really comes down to the whitebox being selected. Facebook does do some hardware design, but it's more of chaining multiple merchant silicon chipsets together or building modular boxes that can house multiple chipsets.

Control-plane/management-plane software is often written by these companies or by contractors.

NICs are another story. MSFT and AMZN both build their own NICs using either FPGAs or custom silicon.

Isn't rushing development in a predetermined time frame one cause of security issues because of lack of time for auditing and QA?

It seems like putting an artificial timeline on feature development would not be in Google or Facebook's best interest.

No, that only pertains to specific parts of the network (which are typically not complex).

Google does operate (lots of) vendor gear, particularly at their network edge.

That may not be up to date:

https://www.blog.google/topics/google-cloud/cisco-and-google...

Reply on news.ycombinator.com