At Cloudflare we use BPF in firewall extensively. We blogged about this in past:

* https://blog.cloudflare.com/bpf-the-forgotten-bytecode/

* https://blog.cloudflare.com/introducing-the-bpf-tools/

* https://blog.cloudflare.com/introducing-the-p0f-bpf-compiler...

We even have a piece of software called "floodgate" which is pretty much a custom runtime for iptables compiled down to BPF. It's using proprietary driver magic to offload the firewall rules engaged during large L3 attacks, and runs them with high performance in userspace. More:

* https://blog.cloudflare.com/meet-gatebot-a-bot-that-allows-u...

More BPF integration in iptables is a very good idea. The interesting bit is how to deal with more complex things like haslimits, limits, ipsets and conntrack integration.