As a man who doesn't drink I can't help but feel the problem here isn't drinking with the opposite sex but drinking as part of a business relationship (outside the Christmas party).

I don't think I would be particularly comfortable at such a meeting; If I did drink I'd not enjoy the act and quickly become introverted (I have a low tolerance, I'm not used to it). If I didn't drink I would feel I'm not committing to the social aspect of the meeting.

What's wrong with a working lunch?