There is standard stuff for this. When you search the internet, be sure to include "embedded" in your search terms.
Personally, I've worked with PC/104 sized systems. These are stackable. We needed a Linux box running on Intel for a project, with up to 16 serial ports. This is all possible with PC/104 based systems.