Actually, no. It only limits the available I/O bandwidth to those GPU(s), that is, data transfers from and to the CPU will be far slower. You'll notice in games when the "loading next level" screen takes longer, that's usually a large geometry and texture copy to the GPU.

As an extreme example, when and where it was/is still viable, cryptocurrency mining operations use external PCIe cases with lots of slots, attached to only a few or even just one lane per case. For their purpose this was still viable, since mining uses very little I/O from/to the GPU, essentially only a small block of setup data into the GPU ("try hashing this block, vary those bytes within that range, report when result matches this pattern", a few kB) and a few bytes of results out ("found a match, varied bytes were as follows").