Actually thinking about it figuring that out would make fission a lot more tenable too
Cooling? an IR barrier in the reactor to stop IR from reaching the cells should solve about 80% of the issue - assuming you can make something transparent at the right wavelengths for the cells - and capable of withstanding the heat of the reactor. The other 20% is already doable with current technology, and we do it from large-scale solar farms to single-family homes - water tubing on the backs of the devices to a radiator or storage tank. Hell, while using them to harvest light energy, that water loop can be used to harvest heat energy, thus improving the overall system efficiency!
These photons can’t usefully be converted directly into electricity with the photoelectric effect, but they sure could be converted indirectly via the same mechanism that the fusion in the core of the sun is reduced to the band that current PV runs at.
Not sure if it’s worth doing that (may well be such a diffuser would be so large it would be easier to do something completely different), but that’s very different thing to your dismissal.
Though you'd need something very different from existing photovoltaics to capture the energy from those x-rays.
It will never produce so much as one kWh of commercial power.
But we already know that fission, too, is uncompetitive, moreso each day as costs for renewables and storage continue relentlessly downward.
Fusion has no chance of ever catching up.
i.e. one needs to prevent the neutrons from escaping or degrading the material the reactor is made out of, so instead of just adding a throwaway shielding layer, would it work to use something that is not only super dense, but is currently a waste product[1], and also becomes more valuable in the process of being used as the shield? Seems like it would be vaguely equivalent to a photoelectric collector for neutrons, but I am not a physicist or reactor engineer.
[1] I do think it's a shame that we stopped building breeder reactors, but that's a separate discussion.
The overarching requirement on fusion neutron absorption material, besides delivery of heat to process steam, is that it needs to produce more tritium to burn. You don't get that if the neutrons are absorbed in something other than lithium.