Semantic caching can reduce Voice AI latency, but it doesn't work well without building contextual awareness in the caching mechanism. The cache needs to understand the context of what someone is saying in order to know if the response can be served from the cache. This blog post describes more about why context is important for semantic caching and how we're building it.
Not gonna lie i've wondered WTF Bing Chat isn't using semantic caching of some form, nope even the fuckin welcome message is slow as shit and seems to be generated each time.