> Because on HN there are only two possible states for comments. Flagged or visible. The upvotes and downvotes just help compute the order.

That's an oversimplification; downvotes may also reduce the visibility of comments by making them an eyebleeding gray-on-white.