Smart speakers are not telephone calls, and even if they were, federal law only requires consent of one party to record (11 states require all-party consent).
In public spaces, you generally lose any right to consent to recording. If you're in someone else's home and they record you with their smart speaker device, then your beef is with that person whose private property you're being recorded on. Same as if they were recording you with their phone's camera in their private residence, vs recording you in a public space.