I do this and it has downsides:

* muting yourself is not clearly visible to the conference app (Your face doesn't get a mic with a strikethrough on the screen). Apps that warn you that your audio device is receiving audio but you are muted on the application level no longer work correctly.

* some apps (Looking at you, Teams) will actually complain that you seem to have deactivated your audio device (of course you did - that's the whole point).