Well this is getting in some rather unfamiliar territory to me, but a function like max(0,x) could be considered a polynomial in the max-plus algebra [1]. In this algebra the values kind of behave like the base-b log of a real number, with b taken to infinity [2]. If you replace b by a really big number you're just dealing with regular polynomials.
In this sense I think you could consider a big neural network with RELU activation a (really big) polynomial.
This is about the extent of my knowledge on the subject though, for more info search for tropical geometry [3].
[1]: https://en.wikipedia.org/wiki/Max-plus_algebra
[2]: https://mathoverflow.net/questions/83624/why-tropical-geomet...