In some cases, I can guess the logic behind the rule, (e.g. "twenty and one second" type idioms) but in other cases, I can't really fathom why it would be so complicated.
1, ends in 1 except 11, 111...: -st
2, ends in 2 except 12, 112...: -nd
3, ends in 3 except 13, 113...: -rd
everything else: -th
(also 0, most people say zeroth but nobody really feels comfortable with it.)
However they came to be, there's no mental load in applying them and nobody who speaks English natively ever much thinks about it.
Come from country who use Type 0 (i.e. no plural whatsoever), English is already complicated and make no sense at first.
On top of this base strata are some more choices. You have some choices to make for 0 (English says that 0 gets a plural form, French believes it to be singular). You can also choose how the rules differ when you get to systematic naming for large numbers--do you lump it in with many, or do you reuse the rules for the small numbers when they show up in larger numbers? Note that systematic naming doesn't have to be based on base-10 (say, Danish), nor does the start of systematic names need to happen on a clean boundary (English starts it at 13, for example).
It'd be hard to make a case for regular number words starting at thirteen. 13, 14, and 15 are all exceptional in that the numbers 3/4/5 are not usually pronounced "thirt", "fort", or "fiff". You could make a case for starting at 16, but even then it isn't normal for the lower place value to precede the higher one. At 21 you see words that are formed in a fully regular way, but the roots for 20, 30, and 50 are still odd, now being "twen", "thir", and "fiff".
From 60 on you can construct number names just by knowing the numbers 1-50 and the place values (hundred / thousand / million / etc.).
In order to call the ordinals perfectly systematic, you need to take the irregular cardinals as a given. If you're having trouble there, you'll have trouble with the ordinals too.