How could managers evaluate my performance, make decisions about where to apply my skills, or understand my value if they knew nothing more than high school math? How would I remain sane with no one around to rubber duck proofs or concepts to, to commiserate or bounce ideas with? Sufficiently advanced technology is equivalent to magic; deskilling of the workforce makes the black boxes even blacker.
Every colleague may not know the precise details of my work, but that this remains a strategic decision and not a limitation due to their lack of skill is essential.