They also have an embedded link to this paper(https://arxiv.org/abs/1810.02054 ) regarding the provable convergence of neural networks by non-convex gradient descent optimization. You are correct, the researchers do no propose an alternative to sgd based training (or at least according to the article). It did seem like it was going toward an alternative training method, perhaps SMO on kernel SVM, but then it pivots to describing the theoretical result for infinte kernel svm from the linked article.