The main problem is still getting strong optical nonlinearity to build optical only transistors. Right now best case 1 in 100 photons will be affected by most "switches". There's been some research using various resonators to enhance the nonlinearities, but that's really the big problem to solve right now.

There's also scaling issues - we have 20nm electrical switches, which is almost 50 times smaller than the wavelength of visible light, so you either need to go to deep UV or confine light to area much smaller than the free-space wavelength, which presents its own challenges.

My personal interest in optical computing is mainly in building optical quantum computers, in which case the cost/size/power of solving these challenges could be well worth it due to the scaling advantages of multi-qubit entanglement. Classical optical computing does not really seem that worth it if we need to build computer chips that are 1000 times bigger and run each operation on average 10^8 times to get a valid result, although the clock speed and power benefits would be nice.