There is already a growing number of people who use tools like super whisper to interact with agents.
At work I'm building a "chief of staff" kind of a voice agent where you simply verbally give it tasks and it goes and gets stuff done.
Also do have a look at the latest announcements from open AI on their realtime voice