There is an assumption in the research community that, yes, stochastic DNA damage and epigenetic drift in tissues has to be important in aging in ways other than cancer, but there isn't any good experiment to demonstrate that this is in fact the case. If anything the available evidence points in the opposite direction, such as mice loaded with a large multiple of the normal level of point mutations that don't appear to suffer any consequences as a result.
When you look at what kills the most people, cardiovascular disease, it all appears to be fairly directly linked to a combination of degredation of elasticity in blood vessel walls (cross-links in the extracellular matrix, a metabolic waste product issue), damaged lipids in blood vessel walls spawning self-reinforcing damage generators (mitochondrial DNA damage, inflammation), and things of that ilk, which lead to hypertension and runaway bad feedback loops in blood pressure and the body's response to same, and then catastrophic failures or blockages in blood vessels. Stem cells and their declining activity are something of a secondary issue in the background, and don't really figure in the important path through this pathology. If your skin and blood vessel walls are loaded up with cross-links that the body cannot break down and increasingly inelastic as a result then it doesn't matter how youthful your stem cells are, you are still on the path to death.