Yeah it's just buggy (same phone here). Works 90% of the time for me (but 100% of the time for my wife, who uses a Pixel 5a).