I can see you would need very solid math and physics to build the next gen 3d graphics/physics engines although I remember an interview with John Carmack where he said that even he wasn't as good a mathematician as people thought he was.
Surely there's plenty of code that goes into games that isn't advanced linear algebra? Like code for networking , game saving/loading , menu systems , IAP payment systems etc. Stuff that probably has more in common with general web programming. It's also possible to buy or get for free various engines to do a lot of heavy lifting for you.
Also some of the most popular games now are 2d or isometric, you could probably write a 2d game from scratch without much more than basic trigonometry , linear algebra and plenty of trial and error.