Lots of things are not especially difficult for someone paying attention to get right. But that doesn't mean they are always got right. Thermodynamically speaking, it is easier to have a digital system that fails to reproduce a sound waveform accurately than one that achieves it, because there is only one way to the right output, and uncountable ways for it to be wrong.
So, even leaving aside gross incompetence (is that smart?), anything that could be sufficiently perturbed thermodynamically, e.g. by age or decay, could throw off your results.
The great advantage of digital electronics is that they make most of the system relatively insensitive to commonly encountered thermodynamic effects, within limits. Most such effects that exceed limits make your thing just not work anymore. (We have all experienced this.) But some can have a more subtle result, some of those without even exceeding those limits.
Fortunately, most of those involve only a few components. Those components are mostly only in your power supply, your amplifier, your DAC, and the clock circuit driving the digital stuff, including the DAC. Of those, the ones that make a difference to sound quality are mostly in the power supply and amplifier, which usually makes sound obviously bad, and the clock, which can make sound subtly bad.
Fortunately, clocks involve very few components and those are not operated anywhere near physical limits, so they rarely go bad. Furthermore, most when they go a little bad don't affect the clock's output in any way that affects what it drives.
Unfortunately, when they do, the effect on the sound you get may be hard to describe beyond "not right". Then, fortunately, you can swap out the whole subsystem with the clock in it to see if that is at fault.
So the most usual danger from a dodgy clock or other source of subtle wrongness is that, as for the case of a needle dragged down a wiggling groove, you might get used to how that sounds, and want it all the time. And that can be hard to match in another system.