No, it’s not against the rules, but downvotes aren’t there to enforce the rules.

It’s just a culture thing. HN tends to be more news and factually focused than reddit/Twitter/etc. People here just prefer to stay on topic, especially since humor is very different between people and can be very hit or miss.