That's not correct; you can say sentences like "The only moving parts in a dishwasher are the two pumps, the sprayer arms, the water inlet valve, and the detergent dispenser"; or "there are only two hard problems in computer science". So "one of the only" sounds just fine to me.