http://news.ycombinator.com/item?id=2619641 (and http://news.ycombinator.com/item?id=2620156 particularly) discusses it.
well phrased description from that article:
"A hellbanned user is invisible to all other users, but crucially, not himself. From their perspective, they are participating normally in the community but nobody ever responds to them. They can no longer disrupt the community because they are effectively a ghost. It's a clever way of enforcing the "don't feed the troll" rule in the community. When nothing they post ever gets a response, a hellbanned user is likely to get bored or frustrated and leave."
If you visit this user's page at http://news.ycombinator.com/threads?id=Humourisok you can see that their first 3 comments had negative karma, after which the hellban was applied. I think it is automatically triggered if you reach a certain negative cumulative karma.