In the rank-1 and rank-2 case these functions can be represented as objects familiar to linear algebra. 3-tensors are not the same as stacked two-tensors, however.
I wonder if "neural networks as ODEs" will catch on. I could use some of my masters work and go get a PhD.
And a neural network is just a stack of linear transforms (the "tensors") and nonlinear transforms (the activations). So in a lot of cases, these "tensors" do do something. Granted the abstraction for storing the active tensors is also a great abstraction for storing high-dimensional data (like a 128x128x4x60 2-second video).
>neural networks as ODEs
You're a few months late :P [0]
Well, in math a lot of things we normally do not think of (or use) as a function can still be considered as such. For example, a vector "is a function." Why? Because it does something (to a covector, or a linear form - which, in turn, is a function, etc. etc.) Even a number "is a function" by a similar reasoning. Objects in math are very flexible conceptually.
Take for simplicity a constant real number. A constant is not a function but there's a function from the reals to a space of real valued functions. Likewise every vector can be converted into a cove tot, but...
You can say these function-valued functions are bijective so you can work on either side of the fence. Your can also work with equivalence classes if you don't quite have a bijection. But a lot of mathematics is about being specific about what we want, even if we're lazy in writing later and smush related concepts together in the same symbol.