Rather, "nine times larger" means multiplied by nine, and "nine times smaller" means divided by nine. This is basic and not particularly awkward, certainly not more so than e.g. positive/negative correlation, or many much more awkward and more common linguistic constructions, IMO.
If you have to edit out words (i.e. context) to argue a phrase doesn't make sense... I am not sure what mental model you have for natural language, exactly, but it certainly isn't a very robust one.
Likewise if I repeatedly subtract a number 9 times to make it “9 times smaller” I can omit the subtracted bit.
If you divide 81 by 9 you subtract 9 from 81 until you get to 0. The number of TIMES you do that is the result “81 divided by 9”.
You chose 81 because it’s the one number that makes your argument appear to work. Try the same reasoning with 80 ÷ 9.