We're already carrying microphones around in our pockets. Do the speaker microphones really make the situation worse? When it is almost guaranteed that there is already multiple microphones present.
I would say they do. For instance, if I got a Google Home it would be the only non-Apple device with a microphone in my apartment, making gov access significantly easier.
Do you really think isolating yourself to the Apple ecosystem makes you more secure? If so: how do you qualify that?