Can you please expand on this and if possible point to some references that'd help in case I want to go down this route myself? Thx.
First, background reading:
https://en.wikipedia.org/wiki/6to4
https://en.wikipedia.org/wiki/IPv6_rapid_deployment
The basic idea is you create a SIT tunnel device and assign it an IPv6 /64 composed of two parts:
1. A network prefix between 32 and 56 bits long. This prefix is the same for all machines in the network.
2. A subnet derived from the machine's IPv4 address, minus the netmask.
For example, if your IPv4 addresses are allocated from 192.168.1.0/24 and the machine has 192.168.1.155, then the network prefix should be 56 bits long (64 - (32 - 24)) and the machine's prefix is `xxxx:xxxx:xxxx:xx9B::/64`.
The Linux kernel knows how to wrap the IPv6 with IPv4 so it can route within your local network to any other machine with a similarly configured tunnel device. If you want to send packets to 192.168.1.200 then they get addressed to `xxxx:xxxx:xxxx:xxC8::1` or whatever, they'll transit the IPv4 network like normal, and on arrival the receiving machine's kernel will strip off the IPv4 wrapper and route the IPv6 locally.
How's this useful? Well, if each machine has a /64 prefix then each pod can be allocated an IPv6 within that prefix without coordinating with other machines. Let's say the pod gets `xxxx:xxxx:xxxx:xxC8::aaaa:bbbb:cccc`. Anything with a correctly configured tunnel and that pod IP can route it traffic, no proxy or iptables needed.
I’ve been meaning for a while now to experiment with this same idea in Erlang. I.e., hack up the Erlang runtime to use an IPv6 address as its PID type, such that each Erlang node running on a machine gets its own /64 subnet to hand out; and each Erlang actor-process on that node gets an IP allocated from its node’s /64 range.
This could just be a way of letting Erlang nodes talk to each-other through tunnels. Or it could be a way of having Erlang “VMs” exposed directly to the Internet as their own little machines.
https://www.cni.dev/plugins/main/macvlan/
Basically just do normal ipv4 via your dhcp server rather than an overlay.
-- Edit
For arguments sake, I just set this up:
root@nas:/opt/cni/bin# ./dhcp daemon
cat /etc/cni/net.d/01-macvlan.conf { "name": "mynet", "type": "macvlan", "master": "eno1", "ipam": { "type": "dhcp", "routes": [{ "dst": "192.168.1.0/24"}] } }
PODIP = 192.168.1.181:8096
Works in my browser; so routes correctly.
Got its dhcp from my pihole.
But in production, I'd rather my ability to launch a new pod not be dependent on a DHCP server being reachable and functional. In that case, this particular trick is rather neat, since assignment of IP addresses is fully static/local (without having to agree upfront what range of IPs each node can use for bringing pods online), while retaining the benefit of everything being directly routable. You can now also run a ridiculous amount of pods on a single node.
Though to counter your point, you don't actually need to use an external DHCP server in my example either, you can just define the block you're giving the server via the macvlan/ipvlan plugin, and I presume, again, it works with both IPV4 or IPV6.
So I guess my wider point is, k8s probably doesn't need to replaced to have the networking work how you like.
https://github.com/aws/amazon-vpc-cni-k8s/blob/master/docs/c...
In the end there is no ideal scenario; boils down to what works best for the use case (or what is the 'least worst' solution). Sometimes it gets you down, but those imperfections can turn a churn job into an interesting one.
We haven't noticed unusual CPU throttling, though we do have some workloads that turned out to be burstier than expected and had to adjust their CPU limits to match.
Note that when it comes to subtle Linux thread scheduling behavior, your experience will depend on which runtime you use, and if using runc then which version of the Linux kernel your workers run. We weren't affected by the CFS bug introduced in Linux v4.18 because we never ran Kubernetes workloads on a machine with the affected kernel, and if a similar bug occurs in the future it might not affect workloads running within gVisor or Firecracker.
Additionally, Stripe has historically cared more about security than efficiency. This lead to an architecture where services run on dedicated VMs, which naturally strands capacity and reduces the impact of bugs that appear at high utilization and/or high core count.
Often see quota's get exhausted through short bursts that don't show up in metrics that then causes CFS throttling to occur even though it looks like the pod is no where near its limit. Also struggled with application startup requiring far more CPU than at runtime leading to ridiculously slow startup times if you had a low limit.
So far our solution has been to just remove CPU limits, but hoping things will get better.
Removing the limits really improved our latency tail, and so far hasn't resulted in CPU saturation at the node level but your mileage may vary
https://github.com/kubernetes/kubernetes/issues/67577
https://github.com/torvalds/linux/commit/512ac999d2755d2b710...
If I recall correctly you need 4.18+ to get the fix.