The problem is that what you think is funny may be uncomfortable to a other people, and for some reason, it's always the funny guy who thinks that the onus is on everyone else to shift their behavior to accommodate his comfort.
It gets worse when half the team is laughing, and the other half are praying for you to be struck dumb already.