Interesting, zer- seems to be similar to Slavic raz- somewhat.
In Russian: davit - press, razdavit - to crush.
Siedlung corresponds to Russian selenie "settlement". Zersiedlung appears to correspond to "rasselenie" (morphologically) and it means more like settlement as a dispersion, movement from a single point in different outward directions.
So I suspect zer- doesn't mean destruction per se, it's just that destruction often involves this movement of parts in outward directions from an original center, which explains the frequent association of zer- with destruction.