If you think how normal fully connected feedforward network learns, its outer products of derivatives. Inference is affine transformations + nonlinearity sequence.
In the rank-1 and rank-2 case these functions can be represented as objects familiar to linear algebra. 3-tensors are not the same as stacked two-tensors, however.
I wonder if "neural networks as ODEs" will catch on. I could use some of my masters work and go get a PhD.
And a neural network is just a stack of linear transforms (the "tensors") and nonlinear transforms (the activations). So in a lot of cases, these "tensors" do do something. Granted the abstraction for storing the active tensors is also a great abstraction for storing high-dimensional data (like a 128x128x4x60 2-second video).
>neural networks as ODEs
You're a few months late :P [0]
Well, in math a lot of things we normally do not think of (or use) as a function can still be considered as such. For example, a vector "is a function." Why? Because it does something (to a covector, or a linear form - which, in turn, is a function, etc. etc.) Even a number "is a function" by a similar reasoning. Objects in math are very flexible conceptually.
Take for simplicity a constant real number. A constant is not a function but there's a function from the reals to a space of real valued functions. Likewise every vector can be converted into a cove tot, but...
You can say these function-valued functions are bijective so you can work on either side of the fence. Your can also work with equivalence classes if you don't quite have a bijection. But a lot of mathematics is about being specific about what we want, even if we're lazy in writing later and smush related concepts together in the same symbol.
Yes, but any parameterized operator needs data for its representation, similar to how a closure is code + data.