To work with U-235 or Plutonium you only need a glove box, but to work with U-233 you cannot have humans physically close to the material, you need waldos and closed circuit cameras and so forth. This naturally increases the cost of working with the fuel. But wait, it gets worse. As I said, gamma radiation is highly penetrating and heavily ionizing, which means that it damages delicate materials quite easily. Especially seals, made out of rubber or silicone or what-have-you, and electronics. This makes fuel cycle handling hugely challenging and also makes reactor construction rather challenging as well.
Now, likely we could overcome these problems but they are nevertheless huge problems.
One of the big reasons why Uranium/Plutonium reactors have caught on is because you can use 1950s technology to build reactors and process fuel. That's not the case with Thorium/U-233.