Then why are you using it? I tried using Gemini once on my Pixel 6. Couldn't play music on Youtube music on verbal instructions, I switched back to Google Assistant. Will try it again after 6 months now. :)
I'm not, I pretty much just accepted that Google doesn't care about usability whatsoever and haven't prompted it in a very long time.
To be clear, the only time I've ever used it was via "ok Google" in contexts in which I'm unable to interface with the phone directly, i.e while driving. If it doesn't work you'll learn that you can't start driving before queueing the navigation anymore. The voice assistant was a nice feature, but not important enough to waste my time trying to figure out which feature they opted me into and how to get back out of it.
That said, at some point it started working better, but there was a good 6-12 months where it was a tire fire.