It's mental labor. If you're a good moderator, there's very little to no emotions in your actions.
In larger, topic-oriented, and/or highly generalist communities, ones that are looser and more impersonal in general, it's been more as you describe.
There's a difference between handling something professionally, or not endulging in it, and not feeling anything at all.
From Wikipedia....
"Emotional labour is the process of managing feelings and expressions to fulfill the emotional requirements of a job.
More specifically, workers are expected to regulate their emotions during interactions with customers, co-workers and superiors."
The mods are expected to regulate their emotions.
Which makes it emotional labour by default.
By this definition my response to you is emotional labor and you owe me for that.