Depending on the handset model, you can open one up and physically remove (or disable) the mic, and use it with a wired headset that you plug in when you want to make or receive a call.
Admittedly it is a bit drastic, but low-tech and very effective.
Most cell phones have pretty low-quality microphones. My guess is that a basic white-noise generator in an inaudible frequency would render the microphone pretty much useless. That being said, there is the problem of then having to maintain the white noise generator.
Stick a neodymium magnet next to it, joking please don't do that
A white noise generator placed over the microphone, perhaps?
White noise can be filtered out if it's not louder than the source sound within the frequency bands of interest over the time interval that matters for interpreting speech. So make sure it's loud, and maybe contour the noise to match the frequency distribution of speech, and maybe make it unpredictably variable over time.
But wouldn't it only have to be loud at the microphone? So a quiet but very close white noise source (e.g. something taped directly over the microphone) could mask a louder but more distant source?
One neat idea from Neal Stephenson's Anathem (with likely analogues elsewhere) is essentially a chatterbox that records people speaking in a conversation, chops up the audio into syllables, and plays those back at random to foil eavesdroppers. This might work better than white noise, assuming you trust the computer that's doing the scrambling!