If you do something like advertise a /32 for each container you can very quickly fill up TCAMs on your network hardware (in particular cheap top of rack switches that are pervasive in data centers).
The entire v4 internet is something like 600k prefixes right now and the routers that can handle that many prefixes at line rate are irritatingly expensive. ToRs as of a couple of years ago when I last tested this would fall over at 1-10k prefixes.
So be careful when looking at BGP solutions because it's very easy to have a BGP topology that doesn't scale, despite it being the exchange protocol for the Internet.
I read your comment as, "Don't use technology that you can misconfigure, because you can misconfigure it!". Well yeah, the same can be said with anything networking related.
Assuming everything is nice and hierarchical, you can easily aggregate an entire rack to a single prefix. Even the shitty ToR switches can usually handle a couple thousand prefixes, which should be plenty if done correctly.
Obviously you shouldn't be advertising /32s.
> The entire v4 internet is something like 600k prefixes right now ...
Just checked my edge routers and it looks like we're up to ~671k prefixes here and that number is still increasing everyday.
At least, that's what you do if you use Calico and want to be able to use hypervisor migration when using it with OpenStack.
Just make sure your TOR's can handle the amount of routes necessary, have a default route from the hypervisor to the internet, and from the TOR to the spine, and have the spine advertise a 0's route. So now the spine is the only place where you need beefier routers that can support more than the TOR's in terms of routes.
With some intelligence in the IPAM solution your host will get a /26 (or a /64) and will advertise that entire range, and only a single /32 is advertised if the VM/container moves to another hypervisor host (to support things like live-migration).
That being said, most TOR's can handle quite a large amount of routes these days. When I was at a telco we had some gear that did up to 128k routes, so splitting between IPv4/IPv6 we had 64k routes each. Which is plenty, even for larger clusters.
Yeah, that's what I meant. I thought that was clear from "aggregate an entire rack to a single prefix" but perhaps not.
When I said don't advertise /32s I meant "globally" and as a general rule (I didn't realize what I said would be taken so literally). There can be exceptions, of course, such as in the live-migration case you mentioned.
Easier said than done. Most datacenters are bit more deliberate about allocating addresses and hand them out in non-contiguous CIDRs. The VLAN mentality is still very prevalent. Getting a /20 at a time is pretty common.
Using overlapping IPs puts you right back into the overlay model.
>Assuming everything is nice and hierarchical, you can easily aggregate an entire rack to a single prefix.
Yes, exactly. The trick then becomes how to you ensure that endpoints that get created within the rack get an IP from the prefix? Romana (the project I work on) does this. It lets you capture your network topology for exactly this reason. This is especially important if/when you must filter routes at ToR.