In the polynomial fit case: it’s pretty clear to me that if the number of parameters is higher than the degrees of freedom in the data, then the model can simply memorize the input.

That doesn’t explain why it generalizes well, though.