This fight is over. You knew it was over the second time you were talking to someone you didn't know well, were about to use the term "crypto", and paused to check in your head how they'd interpret the word. "Crypto" means "cryptocurrency", not "cryptography".
We shouldn't be surprised: there's orders of magnitude more people interested in get-rich-quick schemes than in abstract algebra.
The cryptography engineers will be happier the sooner they let this go and find some new slang for themselves.