There's "programming" that's a single person essentially performing research-level mathematics; there's "programming" that's a thousand people writing to a spec without considering anything about the structure or complexity of their code. Programming is obviously not just math, nor is it no math.
If you become defensive when someone says "programming isn't math" or when someone asserts that it is, you're failing to think critically about the both words they're using and about the assumptions you're making.
Disclaimer: I have a degree in mathematics, and I'm a programmer. I can't think of the last time I used anything from my college years directly in programming itself, but I attribute a lot about the way I code and think about problems in general to having a mathematical background.