We can use Calculus to do so much but also so little…
It is certainly possible that there are complex approaches that the statisticians have not discovered or don't teach because they are too complicated, but they had a big fight about which techniques were provably superior early in the discipline's history and the choices of what got standardised on weren't because of ease of calculation. It has actually been quite interesting how little interest the statisticians are likely to be taking in things like the machine learning revolution since the mathematics all seems pretty amenable to last century's techniques despite orders of magnitude differences in the data being handled.
Circular reasoning: that's true only if the posterior is normal, or if your "optimal" is defined by second moments. In infinite variance cases, the best estimator can be median or an alpha moment for alpha < 2, but yikes the math is much more difficult.
-- A mathematician who has indeed fallen into the beauty trap
That doesn't sound right, it is an error minimising technique. Are we not talking about minimising mean square errors? Why would the posterior need to be normal? And why would optimal need to be defined by 2nd moments?