Like DragonBox's Elements, Algebra and Numbers [1]? Or the ones by DuckDuckMoose, such as MooseMath [2]? KhanAcademy's very first levels might be good as well [3].
With your oldest you might try a game like Junior Catan [4], that than graduates to Catan where you can talk about the probability distribution of the sum of two dice, an essential aspect of Catan.
You could perhaps try a programming language like Scratch. Have a go at some of code.org's "games" [5] and perhaps even MIT app inventor's with your oldest? [6]
[2] http://www.duckduckmoose.com/educational-iphone-itouch-apps-...
[3] https://www.khanacademy.org/math/early-math
[4] https://www.catan.com/game/catan-junior
[5] https://code.org/minecraft
[6] http://www.appinventor.org/content/ai2apps/simpleApps/androi...
That said, if it's fun then there's no harm in it and keep it up. If it ever gets to be no fun though, my advice would be to let it go until they actually need it.
As far as games go, look for something that's fun first and mathematical second. I like pencil and paper games for this. We play them together often while waiting for meals at restaurants or during other down times. Rarely do I call attention to the mathy nature of the game, it's just a fun way to "be mathematical."
The other thing I think is important (that I still have a hard time living up to myself) is to really focus on things that need open-ended exploration. Particularly for children who are in school away from home, instead of trying to reinforce what the teachers are probably already doing a good job of teaching, find ways to get your kids to investigate things on their own. Games with known optimal strategies can be a fun way to get into this because they can then go dominate at school with their new knowledge. They don't get as much opportunity for this kind of thing at school because of the necessity of teaching the mechanics of computation, but keeping their curiosity alive is a great goal to strive toward.
Some resources I like:
Martin Gardner's books
Games from http://www.papg.com/
http://www.expii.com (this is advanced stuff, but can make for fun discussions because some of the situations are neat)
The Beast Academy math curriuculum from Art of Problem Solving (starts at second grade, but when you get there I really like it). http://beastacademy.com
Can you explain this part? A quick google search didn't reveal any relevant games/apps, so the only thing I can think of is that a school of fish can grow exponentially, and that a baby is the logarithm of an adult :-)