People may share empathy between similar feelings, but this does not mean that one person should try to project their way of having emotions onto another person, as if there is a 'right' way to have emotions and a 'wrong' way to have emotions. Emotions are not actions, they are emotions.
For me, as a rape victim, being raped affected my feelings in a way that was traumatic. No one else can tell me what my feelings were from the experience I endured. So now that we can take this as a fact, instead of an opinion, we are getting somewhere in understanding how to define rape objectively, using subjective (but qualified) terms. The reason I feel those feelings is because I was raped.
You don't have to make a judgement call. But some things can only be known on an individual level, and it is up to you to decide whether you believe that person, and that is all you can do as far as 'knowing' anything about it without experiencing it. From that, you can direct your life and how you interact with and understand the world. But that is the limitation you are at.
I do not know what it feels like to be a rape victim who does not feel like it harmed me significantly. I do not know how I would communicate with them because I have never had to. I have never met anyone that has been raped and brushed it off like it was the same as eating cereal that morning instead of toast. So I have no perspective on this.