That said, if it's fun then there's no harm in it and keep it up. If it ever gets to be no fun though, my advice would be to let it go until they actually need it.
As far as games go, look for something that's fun first and mathematical second. I like pencil and paper games for this. We play them together often while waiting for meals at restaurants or during other down times. Rarely do I call attention to the mathy nature of the game, it's just a fun way to "be mathematical."
The other thing I think is important (that I still have a hard time living up to myself) is to really focus on things that need open-ended exploration. Particularly for children who are in school away from home, instead of trying to reinforce what the teachers are probably already doing a good job of teaching, find ways to get your kids to investigate things on their own. Games with known optimal strategies can be a fun way to get into this because they can then go dominate at school with their new knowledge. They don't get as much opportunity for this kind of thing at school because of the necessity of teaching the mechanics of computation, but keeping their curiosity alive is a great goal to strive toward.
Some resources I like:
Martin Gardner's books
Games from http://www.papg.com/
http://www.expii.com (this is advanced stuff, but can make for fun discussions because some of the situations are neat)
The Beast Academy math curriuculum from Art of Problem Solving (starts at second grade, but when you get there I really like it). http://beastacademy.com