(Human) Humor is a far broader spectrum than you describe here too. It can range from "ROFTL because someone accidentally stepped in poop" to deeply layered liguistic jokes like you describe.
So if you're not able to make the right mental context switch at the right moment, you won't get the joke.
All humor? (Honest question)
I was under the presumption that e.g. slapstick or schadenfreude humor doesn't need linguistics, and is therefore seen in many animals. Animals that don't have any form of language but do have rather intricate social systems or even ranks and caste systems. but do have humor. E.g. where breaking or pushing those systems is the humor.
E.g. "boss gorilla slips on a banana peel haha".