If you add multiple GPUs to the same PCIe switch, their bandwidth to main memory of the SOC will be shared. For some use cases that bandwidth was already the bottle neck.
Finally Woz’s design for the Apple II color graphics was based on unified memory.
Dramatic shifts in technology sometimes obviates all our past learning and experience.
PCIe is a tree-shaped network (root is the root hub in the CPU, inner nodes are switches, leafs are devices) with point-to-point links (called lanes) that transmit packets. Bandwidth is determined by the number of lanes that a device's path up to the root hub has at the narrowest node.
Switches are far more complex. The protocol specifies addressing, bandwidth reservations, flow control, priorities and message ordering requirements, which the switches have to handle.
Not a switch thing per se, but also relevant for "single transistors won't do": On the physical layer, each port/link is also relatively complicated to initialize and operate. The transmission encoding, due to the relatively high speeds involved, requires a learning phase where both ends measure the link characteristics and adjust their (relatively complex) signal processing accordingly. The physical layer also handles different link speeds for down-/upward compatibility.
This may explains how they achieve the graphics blitting in the Vision Pro.