Yep, this is what I meant to allude to, and you've worded it much better than I could have.
Perhaps a nice way to say it is that the mathematical objects necessary for physics that I can think of are separable (such as the real numbers). Basically, whenever you have uncountable sets, they come along with some topological structure which must be handled continuously.