It is technically emotional labour though. It isn't physical labour, but it is still draining.
From Wikipedia....
"Emotional labour is the process of managing feelings and expressions to fulfill the emotional requirements of a job.
More specifically, workers are expected to regulate their emotions during interactions with customers, co-workers and superiors."
The mods are expected to regulate their emotions.
Which makes it emotional labour by default.
By this definition my response to you is emotional labor and you owe me for that.
In larger, topic-oriented, and/or highly generalist communities, ones that are looser and more impersonal in general, it's been more as you describe.
There's a difference between handling something professionally, or not endulging in it, and not feeling anything at all.