There is a niche for small, mobile handhelds, with companies such as Retroid and Ambernic making devices with smaller screens. Interestingly even in this niche the proportion of truly mobile vs. portable is seeing the same continuing shift to larger, more powerful handhelds over time.
It seems in all categories people tend to purchase devices with larger screens. I wonder if this is caused by the nowadays much larger proportion of adults using these devices, where the Gameboy, etc., we're seen more as children's toys - meaning smaller form factors and screens might have been a better fit.