The tradeoff is that you can do truly zero downtime configuration changes. Granted, this is important to a very small number of companies, but if it's important to you, Envoy is great.
Where would one take a machine to dynamically reconfigure envoy? How would one configure it?
> The tradeoff is that you can do truly zero downtime configuration changes.
So... just like with nginx?
When I worked in this area a while back - Ingess Controllers and Ingress / a custom type we made because Ingress was too limited.
We didn't use nginx because it would drop requests and mess up connections during certain config reloads. With a custom controller, Envoy never dropped a connection or request we didn't explicitly tell it to (excepting network reliability of course). For context a slow day for us was many billions of requests.
The gateway/ingress controller takes the k8s resources and configures the proxy server accordingly. In some cases additional config snippets specific to the proxy (nginx, envoy, etc) are required, but it's usually just a few lines.
Which http server is used is not that important (the most common ones are all fine), it's more about how well the integration to k8s works.