It's pretty miserable when you're trying to look up a date, and all you get are numbers that requires a conversion step to be readable.
Assuming by number you mean epoch seconds/milliseconds, a number can store the instant in time, but not a TIMEZONE. Obviously you could do bit magic if you shift it over (so you don't mess up sorting), but that's true for any data and a sufficiently large number type.