>> I figured that I don't mind my kids playing computer games as long as they move.
These could also be of interest for inspiration as well - basic idea this Youtuber is doing is to use console gamer joystick re-mapper hardware to convert joycon acceleration readings to button presses/button combos.
Can you Lose weight by playing Breath of the Wild? - Ringfit Controller Mod Explained Version
https://www.youtube.com/watch?v=HYn3pWXMPF8
- Today we talk about the Ringfit Adventure controller mod for Zelda: Breath of the Wild. With this mod you can Exercise and play BotW at the same time.
I made a Ring Fit Adventure Mod for Mario Kart - Controller Bending
https://www.youtube.com/watch?v=CNZSfunw85A
- Today on Controller Bending we mod the Ring fit controller to play Mario Kart 8 Deluxe with. With this Ring-con mod you have to run to drive. Literally. Squeezing the ring-con activates items!
I'd imagine a lot of similar mods could be done via computer vision instead of using joycon accel readings.
I think the key would be to show up to a gaming console as a PS4/XBox controller instead of just a keyboard. I think there are Python libraries to do this [1] or just stick to Steam games that just need WASD controls. This allows you to tap into already huge library of high quality games and just focus on the OpenCV movement-> joystick button remapping part.
[1] Something like this - although it might be Windows only https://pypi.org/project/vgamepad/