No, it’s really not. The rate of increased birth defects for older fathers is quite small[1]. It’s advanced maternal age where you see the multiple orders of magnitude greater chance of serious problems.
It’s fair that my comment is not supported. The evidence is sporadic in both directions in fact. The paper linked is a big study but very weak association results. Here is a good review that includes the most plausible interpretation of the genetics.