When writing feedback, you simply look back at 20 other interviews where you asked the same question, and write "candidate X's performance on this question was was in {top,bottom} {5,25,50}% of ... at this level".
You also have no idea how other interviewers will score, and if your score is 1.5 when everyone else's is a 3.9, it's likely that the HC will ignore you.
The HCs do see multiple feedback from you and your peers, so they build a model of your feedback for themselves: P is a soft interviewer, so we'll adjust his score down but Q is a very hard interviewer so we'll adjust her score up ;-)
What matters is how your scores compare to other interviewers scores on the given slate.
> Not sure how the HCs "adjust" me.
Maybe they don't. Maybe you are the most precise and best-calibrated interviewer on every slate, and never make a bad call. Good for you ;-)