There's a very neat and often forgotten piece of math in learning, VC dimension (unrelated to venture capital):

https://en.m.wikipedia.org/wiki/Vapnik–Chervonenkis_dimensio...

Also, the reason why black box methods are such a big deal now is precisely because controlled/engineered methods turned out to be inferior (obvious example: image recognition; look no further than into the story of the dropout method and Alex Krizhevsky).

Edit: s/basically forgotten/often forgotten/ in the first sentence.