A "cult" is a disparaging term for a religion that someone other than yourself belongs to. YOUR belief group on the other hand is obviously not a cult, but a religion.
(Actually I guess it might be size, but in common vernacular I think the above is actually often the case).
For example, most educated Catholics don't consider the Anglican church or mainstream Judaism or Islam as cults. On the other hand, many Catholics would consider Jehova's Witness and some pentecostal churches as cults, even though at some level Catholicism's doctrines share more bullet points with these organizations than with Judaism or Islam.
Size usually isn't a criterion in the definitions used for groups, whether the disparaging ones or the more objective ones (e.g., in the approach used in Marc Galanters Cults: Faith, Healing, and Coercion.)