I think the complete opposite. A shy, nerdy person who has trouble talking probably isn't truly listening when they're not talking.
A better response might have used a word that wasn't "listening" - like "empathetic response" or something - which immediately dissolves the illusion that you disagree.