Stray thought: Why 4 and 9? Because the joke is funniest if the number is completely ordinary.

0 and 1 are special and so are all prime numbers. 6 is out because it's the maximum die throw. And one figure is more ordinary than two figures, or negatives, or decimals. That leaves 4 and 9.