Because in my experience it is very often like "Call Peter" -> "Today it's sunny in NY".
On macOS it still seems pretty good - I have carpal tunnel syndrome and by Thursday or Friday most weeks I end up using Siri to dictate not code but a lot of conversations in Slack, pull requests, iMessage, etc. In fact, I wrote this reply with Siri right now.
Now it's barely worth attempting, because it gets it wrong more than it gets it right.
But yeah, something about talking to a device which gets things wrong all the time is ridiculously distracting, at least for me.
Sometimes I look back at the road after trying to workout what it interpreted and I feel scared how focused on the phone I became.
Code is much more constrained by language syntax though.
Even for the "call peter" example, while the input is easy, the expected range of inputs that Siri should handle and be able to differentiate it from is huge.
Of course this is still a problem for e.g. defining variable names, where you could say anything.
Are either of those companies investing particularly heavily into voice agents? Certainly neither of them has anywhere near the kind of power of something like Copilot.
Also, a general agent is way different from one that's specific to writing code.