You can run full blown Linux efficiently at 500MHz or 600MHz processors like STM32MP1 processors, powered by AA batteries or other small battery packs.
There's also SAMA5D2, and a few other competitors in this space (both above, and below, the STM32MP1).
When we're talking about "consoles", that's "plug-and-play executables", meaning you now want a proper compile / library -> ELF + loader == Linux kernel, security, etc. etc.
Besides, a DDR2 chip gets you like 512MB of RAM for $4 and easily fits within the power-constraints of AA-batteries. There's very little benefit to going to the microwatt-scale devices like STM32 Discovery.
----------
Microprocessors for the win. Entry-level MPUs exist for a reason, and there's a ton of them well below Rasp. Pi in terms of power / performance.
There's many at the 2D level of graphical performance, but 500MHz is still a bit low for this. You'll probably want to reach into faster 1000MHz / 1GHz MPUs and push into STM32MP2 if you're reaching into 3d levels of performance. (Which is beginning to look like a cut-down cellphone chip really)