I am going to be obtuse and say that since Bayes theorem is a non controversial rule about conditional probability, if you interpret your NN as a probability distribution over outputs that is updated by data, it is always able to be interpreted as "Bayesian" and can be a helpful way to examine what your implict priors are (i.e. via architecture or regularization terms) to see if they are reasonable for the problem at hand. It is no surprise that weak uninformative priors are sort of useless. Explicit priors shine when you know something about the actual problem (say some moments or some invariances in the problem set up).