The core observation here is that NN nonlinearities can be approximated by polynomials. And sure, that's true, but especially with low degree polynomials the approximation might suck, or have very different properties from the actual function in question. In particular, RELU activation doesn't explode towards infinity the way a high degree polynomial activation would.

OTOH, no one cares what the nonlinearity actually is so long as the net trains, and there's a lot of effort to keep layer inputs in a neighborhood near zero (batch normalization), so polynomial explosion may not be such an issue.

Feel like I would like a more serious comparison of their model's results with best of class NNs. I'm suspicious that their NN character detector was basically failing...