If racing frames didn't break from extended use, then they could be lighter. You want them to be just barely strong/stiff enough.
A few grams of weight does not have enough impact TT bike performance enough to make it worth shaving 100 grams if it risks reliability.