English is really consistent here. 4321 is four thousand three hundred and twenty one. Probably the biggest inconsistency in English are the numbers 11 to 19.
Many European languages are not. Often with numbers between 11 and 99 are a weird special case. German puts the single digit first. As another poster mentioned 23 is "dreiundzwanzig" (literally "3 and 20"). French is worse where 97 is "quatre-vingts dix-sept" (literally "4 20s 10 7").
These are just the last remnants of a pre-decimal ("vigesimal" specifically) counting system.
I don't really see how weird rule exclusions on certain numbers would give anyone an advantage in math, particularly because even though the word means "4 20s" when you learn it, it's just a word and it may well just lead to questions like "why isn't 40 2 20s?" There is a cognitive cost with exclusions.
I remember reading a study once about how verbs became regular in English. There was a mathematical relationship between how often the word was used and how quickly it became regular. The less used, the quicker it happened.
This is why "to be" is irregular (as its the most common verb) whereas most other verbs (with <100 exclusions) follow a simple pattern.
It's probably the same with numbers. The numbers less than 100 are used far more often so their irregularity is preserved far longer than the less common numbers above 100.