This is a common criticism. However, almost all ML methods have some built in heuristic choices, that are the result of finding something that both works and is mathematically nice. Each of these choices restricts us to some family of functions where it's hard to justify why it's really relevant to the problem at hand, e.g. convex loss functions (l1, l2, ..), convex regularizers (l1,l2,..), gaussian priors, linear classifiers, some mathematically nice kernel functions, e.t.c. In the end, people usually statistically estimate the performance of the methods and use what works.