They can hear everything you say at any given time, even if your phone is off. Only removing the battery can prevent this. If you think this is a stretch, you are sorely mistaken.
Then there's the issue with the towers/lines being jammed by every phone constantly transmitting.