If there was enough soap residue to matter you'd notice the taste.