Parameter instability does not worry a machine learner as much as it worries a statistician. ML folks worry about output instabilities.
The current understanding goes that this overparameterization makes reaching good configurations easier while keeping the search algorithm as simple as stochastic gradient descent.
(Asking specifically about time series models and such.)
https://arxiv.org/abs/1803.03635 ( you can follow up on semantic scholar for more)
Selecting which weights to discard seems as hard as the original problem. But random decimation, sometimes barely informed decimation have been observed to be effective.
On the theory side now it's understood that in the thicket of weights, lurk a much much smaller subset that can have nearly the same output.
These observations are for DNNs in general. For time series specifically I don't know what the state of the art is. In general NNs are still catching up with traditional stats approaches in this domain. There are a few examples where traditional approaches have been beaten, but only a few.
One good source to watch are the M series of competitions.
Latent space clustering is about as good as it gets imo, and in my experience, that's fairly stable for individual implementations (but not necessarily across implementations for the same model, for various reasons), but it doesn't tell you anything about the meaning of the parameters themselves. If the model is well calibrated, you can validate its performance and it becomes explainable as a unit.