There are lots of logical reasons for this reaction, and I will list some of them briefly. 1) Power of VC network is implicit threat to 'play nice' and go along with it, even if VC in question has no intention of being nasty in any way. 2) The power imbalance already implicit in VC/founder relations makes 'asking me out' incredibly tense. 3) It would make me doubt myself- "Is every other VC I am talking to (since they are 90% men) just interested in me for that reason?".
So follow the OP's advice. And just don't do it.