So what if the panel actually even disabled the mic preamp on the board? You could have a program/virus/trojan that silently re-enabled it behind your back, without you knowing. So I'd say this is nothing to be concerned about, because every device you own potentially "listens" with its microphones all the time.
The advice with plugging a jack into the line-in port is good -- if you're sure your hardware hasn't been tampered with, this should physically disconnect the microphone.
Obviously if you're really worried about your computer listening, you should rip it open and physically cut the microphone connection.