Yes. As a married, white, heterosexual male, being seen having a dinner alone with a single female (or even lunch) would be enough to cause questions from friends. So it's not worth it.

And I'm no one, with no fame or high position.