I'm going to second this, massively. Games are both serious software (take a look at how big of a chunk of the entertainment industry they represent), and often make for difficult problems. Your interviewer was just an ass.

I'd second this. I don't game, and have never had any interest in developing games, but there's certainly nothing wrong with it. I've always seen game development as a positive when hiring. Keep it up.

Another non-gamer agreeing that games are seriously tough software to program correctly. One of the relatively few places in software engineering where you really need to use a lot of applied math.