Just spitballing here, but if your interface to these things is USB-C, you should be able to boot them off an image that has a standard SSH key and then you can get in and ID them from a serial number or MAC address. I don't see the identification part as being a huge part of what's gained with the 1:1 configuration.
Is video forwarding part of the product offering, or simply considered required for management functions?
Either way, I probably agree that a raspi per unit probably makes sense at a scale of a few dozen racks, but it would be interesting to do the math on when it would be price-efficient to have a 1:n management node scheme. I don't imagine there are many USB-C hubs that support being display sinks on the downstream ports (if that's even possible at all) but perhaps you could use an FPGA to synthesize a small ARM core with a bunch of native USB-C interfaces capable of doing it?