"to" is often reduced to something like "tuh", whereas "two" never is.

Except in deliberate word-by-word pronunciation, "one two three" and "one to three" sound different in native US English.