Projects like this are some of my favorite uses for single-board computers. Another one is Bare Metal C64, which aims for low-latency vsynced Commodore emulation on the Pi: https://accentual.com/bmc64/
There is one small difference, BMC64 uses Circle and circle-stdlib to produce a bare metal image and does not rely on a Linux distribution: https://github.com/smuehlst/circle-stdlib