I mountainbike, started with trail and am now doing more XC. When I ride either type during training (or just fun rides), I always wear knee pads. But when I race XC I don't, even though often we'll do the same downhill sections (not black diamond obviously, but certainly rocky blues).
I think it might be that, at least in racing, knee pads restrict your movement a little. Though I've been wearing them for so long I don't even notice them anymore, and certainly if you wore knee pads on a road bike they would be less restrictive than even lightweight MTB knee pads.