I'm frequently in a room with people that know less about mathematics and programming than I do (and let's be clear, I don't regard myself as exceptionally talented in either field). It's not surprising, that's usually the reason I've been asked in to consult.
The great joy is discovering what these people know more about than I do: electronics, cartography, avionics, wet film processing, fire fighting, violin making, linguistics, ... it's always a surprise finding out just how much some people know about their particular interest and how well they can make you appreciate it.
I'm get a little tired of hearing "the IT Crowd" cackle on about how stupid their users are, maybe that is true for them but it is possible to go out and find end users that stretch and challenge you as a programmer to meet their needs.