Exactly. The mathematical concept behind this is the idea of compactness[0, 1]. A compact manifold always has finite volume. In the same way, a compact dimension always has finite length and this is what is meant here.[2]
Note that the converse does not hold: Not every dimension of finite length is necessarily compact. (The mathematical reason being that the metric tensor could become smaller and smaller ("fade out") towards the infinite ends.) But for the purposes of talking about string theory, you can usually equate the terms "finite length/volume" and "compact".
> Is "small" then always finite, but large can be large-but-finite or inifinte?
Yes, "small" is always finite (compact). As for "large" dimensions, I would say that's a matter of terminology and physicists tend to not be very precise with their terminology. It can both be large-but-finite (compact) or infinite (non-compact), depending on the context.
[0]: https://en.wikipedia.org/wiki/Compact_space
[1]: https://mathworld.wolfram.com/CompactManifold.html
[2]: This is also what string theorists mean when they "compactify a dimension": They take this infinite dimension and "wrap it around" a circle (of a given circumference) and end up with a cylinder or some more complicated object (depending on what they started with).