Time is a number. At least, it makes sense to subtract them (I guess you can say that time is a one dimensional affine space).
For uint, you actually have x + y = (x + y) mod MAX_UINT.
For float, sometimes x + 1 = x.
The issue is not that date is not a number, but that people poorly reason about common cases of date arithmetic. Similar stuff comes up for floats and ints but they're just easier to reason about.
Would you spell this out for us?