Hacker News Reader

Comment by yrro | Hacker News Reader

I think the article could do with a bit more information about _why_ you'd want to use macvlan vs a bridge.

macvlan lets you offload switching to the network--so more of your CPU power can go towards running VMs. Since all traffic, even that between two VMs, goes via the external switch, that external switch can also perform auditing, accounting and policy enforcement. If you're a big enterprise with lots of expensive switches then you probably want to use them for these sorts of things, rather than push the job down to the hypervisors.

View on HN

In my experience with said expensive hardware, it greatly depends on your use-case / scale.

You have to pay attention to stay within the limits of the hardware the switches (TCAM, etc...). And correct me, if I am wrong, but most NICs are fairly constrained in their hardware support on the numbers of MACs they can filter for. And those two combined, you might end up with a lot of flooding, which needs to be handled by the CPU again.

You also have to rely on the implementation of your vendor. And at least with one major vendor, who shall remain nameless, it is a major effort just to get acknowledged that there is a bug.

The point about number of MAC filters is fairly moot as when you do bridging in hardware you are not going to utilize any kind of hardware MAC filtering capability anyway. Also the issue of flooding tends to be solved by the physical switches in the network.

So, we are in agreement in that point: if you are going that way, you are relying on the physical switches.

I'm not sure about the problem being solved, depending on the scale.

If you are with your use-case (number of rules, macs, five-tuples, ...) outside the capabilities of the hardware, then, in my experience, it breaks down quickly. (I.e. flooding, or becoming unresponsive to configuration commands and needed to be _power cycled_, ...).

And those are switches costing easily five figures, and from two different well known vendors.

If you have practical experience with a setup, which scales more gracefully, please share.

But I'm sceptical that it exists, considering Google, Facebook, AWS all went with custom hardware.

I'm wondering, if at a certain scale of virtual NICs/flows per port, you are not better off with a hybrid approach, where software is part of a decision hierarchy. E.g Openflow.

No practical experience though.

> macvlan lets you offload switching to the network--so more of your CPU power can go towards running VMs. Since all traffic, even that between two VMs, goes via the external switch, that external switch can also perform auditing, accounting and policy enforcement. If you're a big enterprise with lots of expensive switches then you probably want to use them for these sorts of things, rather than push the job down to the hypervisors.

The current trend is to push that to DPU/IPU-style NICs which have their own cores. Previous generation thing was to reduce CPU load on hypervisor by offloading the packet processing so (smart)NIC.

One use-case I’ve seen for MAC VLANs is specialized audio transmission (AES67 maybe?) where data is sent over multicast.

Never really fully understood how it worked, but when developers wanted those applications to be hosted in containers, there was a lot of talk about MAC VLANs :-)

Interesting. Could you or someone else say what benefits does MAC Vlan offer over IP multicasting?

So it's kinda like doing VFIO, but with less things handled by the NIC, but externally from the switch point of view it looks like the same (several MACs going out of a single port), right ?

Reply on news.ycombinator.com