Numbers may be (or not, it may depend on our biology) a good initial concept, but maybe something else may be developed, something more "correct" to deal with the tasks of describing reality, something like metric vs imperial units.
The idea of integrating time to your vision of reality in the way it is used in the Ted Chiang's The story of your life (the movie Arrival was based on it) could be a good approach.