Although of course that behavior may be a signal that the model is sort of guessing randomly rather than actually producing a signal.
The LLM isn't performing the desired task.
It sounds possible to cancel out the comments where reversing the labels swaps the outcome because of bias. That will leave the more "extreme" HN comments that it consistently scored regardless of the label. But that may not solve for the intended task still.
The LLM isn't performing the desired task.
I wouldn't treat the output as complete garbage, just because it's somewhat biased by an irrelevant signal.