Beyond L2 Loss – How We Experiment with Loss Functions

In Bayesian inference this isn’t actually as helpful as you make it sound. In some special applications there have been physically motived (even discovered) prior distributions, like Weibull, Gumbel or Rayleigh distributions.

But almost never do you start from freedom for arbitrary loss functions and work backwards to what prior they would imply. That’s mathematical curiosity that doesn’t connect to understanding of the problem domain.

More usually you will put simple, known form priors on the quantities you need to model, and then abstract the parameters of those priors into hyperprior distributions, and just keep doing this process until the parameters you end with are so abstracted from how they influence the inference that it will be perfectly valid to assume uninformative priors at that stage.

You would only adjust the functional form of your intermediate, parameterized priors if doing this and carrying out model fitting ended up with very poor model fit or poor posterior predictive checking, in which case it would be the aspect of poor model fit that informs you about problem structure and leads to a revised prior, not the other way around.

Even when there is poor model fit, changing from simple priors / uninformative hyperpriors won’t be a first reaction. You might try data cleaning, transformation of variables, dimensionality reduction, collecting more data, changing the core likelihood function of the model, and many more things before having good enough reason to think that manipulating prior distributions is going to be important for the solution.