Fusion produces neutrons, not photons.
These photons can’t usefully be converted directly into electricity with the photoelectric effect, but they sure could be converted indirectly via the same mechanism that the fusion in the core of the sun is reduced to the band that current PV runs at.
Not sure if it’s worth doing that (may well be such a diffuser would be so large it would be easier to do something completely different), but that’s very different thing to your dismissal.
Though you'd need something very different from existing photovoltaics to capture the energy from those x-rays.
It will never produce so much as one kWh of commercial power.
But we already know that fission, too, is uncompetitive, moreso each day as costs for renewables and storage continue relentlessly downward.
Fusion has no chance of ever catching up.
i.e. one needs to prevent the neutrons from escaping or degrading the material the reactor is made out of, so instead of just adding a throwaway shielding layer, would it work to use something that is not only super dense, but is currently a waste product[1], and also becomes more valuable in the process of being used as the shield? Seems like it would be vaguely equivalent to a photoelectric collector for neutrons, but I am not a physicist or reactor engineer.
[1] I do think it's a shame that we stopped building breeder reactors, but that's a separate discussion.
The overarching requirement on fusion neutron absorption material, besides delivery of heat to process steam, is that it needs to produce more tritium to burn. You don't get that if the neutrons are absorbed in something other than lithium.