That doesn't seem right. Sperm and eggs are created by cell division too. In some sense you come from a line of cells that extends long before your birth. If cell division "wore out" after some number of divides you should be already worn out.
I haven't been able to find a paper describing a mechanism, but my literature review tends to specialize in hymenopterological ethology, so I don't know that I would find such a paper even if it exists. The correlation is uncontroversial, though, and I suppose it might not be a surprise to see it demonstrated that the cause is in fact the accumulation of errors in the genome that's half-duplicated during spermatogenesis.
I had an internship at the New York State Psychiatric Institute (before I moved into software development). Paternal age was used as one factor in determining if people were at risk for psychotic breaks (schizophrenic prodromes) by one of the assessment tools they developed.
Individuals do get less good at it over time, though. The structures described earlier as "shortening" are called telomeres [1], and they do actually do that, as a result of DNA copying being a physically somewhat destructive process. They're purely structural rather than informatic, though, and everyone starts out with a fresh set.