As crude as it may seem, rules engines and long sheets of If this Then that heuristics tend to solve more complex systems that can't be statistically fit, but I feel machine learning is slowly approaching a point where it can deduce those If Then statements. The main problem is that as humans we cannot comprehend the wealth of connections that might lead to a certain heuristic, so it is very difficult to tell a modern AI that it is wrong in a more abstract or pattern-based sense, and these approaches tend to collapse back into statistical data fitting. I can easily communicate with you if your understanding is wrong, but it is hard to do that with an AI, mostly we can either fix the data, or adjust the training parameters and wait. That is why it seems to rely on all-or-nothing, as there isn't much control on the iterative learning process on a network apart from startup options and testing after the fact.
To me, the real breakthrough will come by creating models of abstract thought. Some models exist as heuristic systems, which require an exhaustive set of rules to model various aspects of thinking, and some are probabilistic in nature and can be taught to a machine learning application, but neither of these model everything, nor do they do all of their topics equally well. Whichever paradigm can come up with a robust model first will likely come up top in the AI paradigm wars, but it could be tomorrow, could be next century.