One can generalize this approach further to take any given loss function, not just a p-norm, and ask what statistic is achieved by minimizing loss.

A field that studies this is called property elicitation, from the idea that the loss function elicits a certain kind of prediction be it mean, mode, etc.

This field actually originated with human experts like weather forecasters - what is a loss function (or “scoring rule”) that incents them to predict accurately? Squared loss is good if by accurately we imply we want the mean.

One fact people find surprising is that squared loss is not the only one that elicits the mean. There is an entire family of losses that do, including KL divergence.