Kernel methods are linear methods on data projected into very-high-dimensional spaces, and you get basically all the benefits of linear methods (convexity, access to analytical techniques/manipulations, etc.) while being much more computationally tractable and data-efficient than a naive approach. Maximum mean discrepancy (MMD) is a particularly shiny result from the last few years.
The tradeoff is that you must use an adequate kernel for whatever procedure you intend, and these can sometimes have sneaky pitfalls. A crass example would be the relative failure of tSNE and similar kernel-based visualization tools: in the case of tSNE the Cauchy kernel's tails are extremely fat which ends up degrading the representation of intra- vs inter-cluster distances.