It's interesting, because many of your problems there are relatable to the simpler deploy discussed by the parent. I'd be no wiser debugging your bespoke ansible script, and likely neither would you, if not for the fact you've written it.

Don't get me wrong, debugging overlay networking issues isn't something to love, but it's also not all that complex:-

- There's a worker daemon on every box that manages the local configuration, whether thats IPtables, IPVS, BPF or something else. There may be a seperate worker for service IP addresses than pod IP addresses.

- There's a controller that does the actual figuring out what things should be doing and lays out the rules for the workers. This might include network policy controller, but this might be in a seperate daemon.

This setup enables Service IPs, Pod IP addresses & Network Policy.

Obviously in ansible you can just write your own firewall rules, but as soon as you step away from running every app on every box, you'll either be relying on something as complex (but managed by someone else) like the cloud providers SDN, or you'll need to run your own system that does the same.

As much with anything, it depends what you're doing, but I like auto recovery, app level health checks, infrastructure as code, namespaces, resource quotas, and don't want to force my dev teams to couple their network policies with infrastructure details, so I'm fairly happy with the abstraction.