I think silicon photonics are for directly connecting chips on board with other peripherals, in order to save energy and speed of light.
The silicon real estate being used for interconnect fabric PHIs is becoming significant and, unlike compute, do not scale well with node shrinks. On-die photonics means you can cut out a big chunk of your PHIs and avoid the the latency of going to NICs. The PIUMA architecture even used on-die switches to avoid having to use external switches.
https://www.tomshardware.com/news/intel-demoes-8-core-528-th...