Hacker News Reader

Comment by smcleod | Hacker News Reader

Slightly aside of BGP on an internet scale - We’ve recently switched to using BGP internally within our network for kubernetes + load balancer (nginx) routing, (announcement / discovery) amongst other things and I can tell you - it’s one of the best moves we’ve ever made. The simplicity, performance and reliability is brilliant.

https://www.projectcalico.org

View on HN

We did the same thing but opted for the much simpler kube-router, which is more k8s specific and doesn't have nearly as many moving parts (easier to troubleshoot as a result). Bonus point is uses LVS for service proxy load balancing:

https://www.kube-router.io/

https://github.com/cloudnativelabs/kube-router

Depends on your use case, the network policy features in Calico are very important for many real-world use cases.

What does calico do (Ingress and Egress policy) that kube-router does't?

https://cloudnativelabs.github.io/post/2017-05-1-kube-networ...

OT: is this really "simple" to setup or do you mean that once it's running that it's simple? or what do you mean with simplicity. I actually looking forward to use k8s in our network, too. But I think the simplest solution for HA k8s is using keepalived paired with nginx/haproxy on the master level and add a keepalived-vip for every service.

(at least for our 3 node master, 3 node worker setup)

(I evaluated the calico + bgp option, too but it looked way to complex, as a k8s starter)

Yes, it was very simple to setup, deploy and understand on an ongoing process, BGP is incredibly simple by design.

I’m heading to bed now (AEDST), but check out:

- https://docs.projectcalico.org/latest/getting-started/kubern...

- https://kubernetes.io/docs/tasks/administer-cluster/calico-n...

And there’s plenty of other resources out there. IMO if you’re designing a highly available platform, BGP is one of the simplest parts of that complex system.

In my experience you’re likely to run into much more finicky problems with higher layer systems including failover delays, reliance on casting systems and state drift.

Kubernetes is very complicated, but BGP is not. If the curve for BGP is too steep, you can do OSPF inside your racks with BGP announcements to spine. This is pretty common as OSPF can use area 0 with little to zero configuration. BGP leaf to spine is standard is layer 3 networks.

As an alternative approach may I suggest to look at the way kube-router does. It basically leverages BGP and ECMP to provide HA.

https://cloudnativelabs.github.io/post/2017-11-01-kube-high-...

Do you have loops internally?

I would think people deploy BGP to automate routing and routing policies, not primarily to avoid loops.

Indeed, completely for fast, simple routing (rather than higher level, more complex failover).

Slightly off topic, but in my experience most (general) internal networking loops stem from mis configured devices bridging networks (desktop level, network device level or VM / SDN)

I may be old, but if you have a network loop inside your own infrastructure wouldn't it be way more sane to handle that on layer 2? for instance with properly configured spanning tree?

Spanning tree just prunes links from the network to eliminate loops. Sometimes the link that got pruned happened to be the fastest path from point A to point B. So, sometimes you can get a more efficient network if you leave those links in, but that requires more sophisticated routing.

A routing protocol will avoid loops more efficiently

Routing is a Layer 3 issue. Loops are a Layer 2 issue.

Multiple paths on a Layer 3 network provide redundancy when used with a routing protocol such as BGP.

Multiple paths on a Layer 2 network provide redundancy when used with a protocol such as LACP. Without proper design, 2 network segments connected with multiple links will trigger Spanning Tree and shut down one of the links. And depending on the configuration, it may or may not unshut if the active link pair goes down.

Also, it's allows loops to be used as multi paths from different points in the net. Say you have a rack at south and north and there is a loop to a rack in the middle. For failover, you'll take any path that is up. But for normal operation, you'd prefer to take the shortest path.

Layer 2 technologies like MC-LAG do that as well (though I agree that it's better to route, if possible)

Reply on news.ycombinator.com