"Physics is to math what sex is to masturbation" -Feynman

That said, I'm not sure that he actually held contempt for math. It's fairly essential for theoretical physics.