Would it really be that bad? If the devices were well behaved (ie. not too noisy, no gratuitous ARP etc) and the application could assume that most of the time either zero or no devices will be communicating, would it be that bad?

This is something I might test myself. I have a couple of audio devices that will never be both "active" at the same time. In my current layout I need to run either two cables or just another switch and it just seems a waste. I wonder if I can buy daisy chained cable so I don't have to make one...