The cool thing about graphics programming is that often times you aren't even doing "linear algebra" strictly.

A lot of the math for 3D graphics programming uses the concepts of affine spaces/transformations since standard 3x3 matrices don't have enough information to support translation/projection. I had no clue about this branch of math at all until I started learning graphics programming--in fact I think graphics programming requires you to learn the most math of just about any discipline of computer science outside theoretical computer science. The amount of math you need to truly "understand" path tracing is immense.