In Chinese numbers are also prefixed to measure words or classifier. eg; "How many brothers do you have?" No one will answer "m" but "m'go". So, I don't believe in colloquial Chinese that a pure number is a word. You are welcome to disagree.
By the way, it is "m" for 5 in Cantonese. Unless you were talking about Shanghainese, but then is "n". "ng" is a common starting consonant in Cantonese, like "ngao" (cow) or "ngo" (I). but it doesn't have a standard meaning.