https://www.haproxy.com/blog/dns-service-discovery-haproxy/ (which is a link on the OP) says TTL on SRV records "is ignored by HAProxy as HAProxy maintains its own expiry data which is defined in the configuration".

Is this true for A records too?

If so, neither haproxy nor nginx expire cached A records.

Nginx Plus does, and a few nginx plugins do, however.

https://github.com/airbnb/synapse is a process that polls DNS, and updates haproxy config accordingly and SIGHUPs haproxy I've used synapse to solve this issue, but it's a moving piece I'd rather not have involved.