Typical restart-all-the-things usually made it go away. But it wasn’t unusual for 500ms of latency to slowly build up during a 30min call. Unfortunately I have nothing more useful to add, the issue resolved itself before I could track down a definitive cause.
Stop using it.
Without even looking at your setup, I would bet $100 minimum that it's Bluetooth latency. It adds a lot of latency, 500ms is not unusual, and many folks have no idea that all that latency is really just the last 18 inches. This is why you're seeing more and more cases where people are using good old iPhone wired earphone for conference calling, especially when skyping a TV interview.
How are you measuring? Half that is considered high-end from what I'm reading[1], and AirPods Pro apparently reduce that to 150ms.
Bluetooth definitely is the issue with 95% of these latency issues.
Crazy that airpods have that much latency. I have bluetooth in the car and it has way too much latency - I thought Apple had improved it to not be noticeable but I never checked the numbers.
Incidentally, for playing midi instruments you generally want things to be below 8ms to feel natural. 150ms is an eternity.
Is the lipsync ok if you watch a video on an iphone with airpods?
For video they compensate for the latency by displaying the visuals slightly delayed (this has been done for over a decade even way back on feature phones).
Even game consoles have the option to do this since some TVs/receivers have audio or video latency.