This is a perennial problem for the handful of Pi's I have deployed around the house (homebrew DAC, weather station, boiler manager (wip), workbench terminal). They're all 'purpose built' and sit almost entirely idle while fulfilling a single, usually non-interactive task. And after 18 months or so, they all start failing the same way:

* Random userspace crashes (most recent one was a reproduceable crash in the bowels of the python runtime) happen with greater frequency

* (Usually in the process of debugging the above) strange filesystem artifacts show up (files disappearing after reboot, etc)

* Booting ends up taking 5+ minutes, if it succeeds at all.

The solution to the above ends up being 'rebuild the box on a new SD card'. This is usually on 'brand name' (Kingston, etc) SD cards bought through proper retail channels, so while they're not high end cards, they're at least not (likely) counterfeit either. I still don't have enough data to know if simply dd'ing the old card over is sufficient, or if a full ground-up rebuild is sufficient.

Would love advice on this, especially around brands of cards with known good lifetimes in similar environments.