The microphone is a hard one. Even if you cover the holes in the casing where the mic is located the sound still travels just fine via the keyboard.
IIRC when a photo came out of Mark Zuckerberg covering his webcam I noticed he covered his mic port with tape as well. He should have checked the volume levels from his mic - it would still pick up everything just fine. In fact I think the tiny mic holes are mostly vestigial.
My PC doesn't have a mic plug and the headphone one is only out, no in. With two mics (internal and USB) I can switch among them by software. I'm pretty sure I could do that with an analogue mic.
I wonder if an app could do the same on a phone, switching from the outside dummy mic to the internal one. You'll definitely notice that if you're calling somebody, but if the app only listens when no other app is using the mic, then it's easy to record audio most of the time.
Maybe a solution would be no integrated microphone, a small low profile microphone in the laptop box to be connected through the laptop mic jack.
Depends on the make/model of computer - current MacBooks use a lot of glue to hold components together, making minor hardware tweaks a bit of a gamble. It's not impossible, just not as hassle-free as it used to be.
> or at the OS level
If your computer has a rootkit then doing it in software is no good.