Hmm - a counterargument would be to substitute "literacy for "mathematics" in the argument; literacy was elitist in its core historically (and still is in certain societies, unfortunately) - does that imply that learning to read is irrelevant for most people?