I thought Word 2 Vec isn't "Deep Learning" as both CBOW and skip-gram are "shallow" neural models.
Still, interesting blog post, worth reading and googling for more information. FWIW I just use the deep learning definition of "neural net or representation learning research since 2006" and find it fits better.
Logistic regression with regularization is fairly new? 'Pioneered' by the same people as deep convolutional neural networks? Are you certain about this?
Drop out itself IS fairly new[1]. As well as its newer cousin drop connect.
I agree neural nets themselves are basically just a crazier parametric model. Many of the things we do to modify the gradient are applicable to logistic and other simpler regression techniques as well.
[2]: https://cs.nyu.edu/~wanli/dropc/dropc.pdfhttp://www.cs.toron...