By the way, why don't we just call them "arrays"?
Perhaps I'm missing something, but (practical) deep learning seems to have little tensor algebra going on, just operations on slices of arrays.
Perhaps I'm missing something, but (practical) deep learning seems to have little tensor algebra going on, just operations on slices of arrays.
It could also be dated to earlier work on computer vision where tensors were used to store a bunch of spatial coordinates so it might be that back then it was even more aligned with their mathematical definition.
That said the grief doesn’t seem to be with the naming convention but rather with how tensor classes are designed in common DL frameworks.
If you think how normal fully connected feedforward network learns, its outer products of derivatives. Inference is affine transformations + nonlinearity sequence.
In the rank-1 and rank-2 case these functions can be represented as objects familiar to linear algebra. 3-tensors are not the same as stacked two-tensors, however.
I wonder if "neural networks as ODEs" will catch on. I could use some of my masters work and go get a PhD.
And a neural network is just a stack of linear transforms (the "tensors") and nonlinear transforms (the activations). So in a lot of cases, these "tensors" do do something. Granted the abstraction for storing the active tensors is also a great abstraction for storing high-dimensional data (like a 128x128x4x60 2-second video).
>neural networks as ODEs
You're a few months late :P [0]
Well, in math a lot of things we normally do not think of (or use) as a function can still be considered as such. For example, a vector "is a function." Why? Because it does something (to a covector, or a linear form - which, in turn, is a function, etc. etc.) Even a number "is a function" by a similar reasoning. Objects in math are very flexible conceptually.
Take for simplicity a constant real number. A constant is not a function but there's a function from the reals to a space of real valued functions. Likewise every vector can be converted into a cove tot, but...
You can say these function-valued functions are bijective so you can work on either side of the fence. Your can also work with equivalence classes if you don't quite have a bijection. But a lot of mathematics is about being specific about what we want, even if we're lazy in writing later and smush related concepts together in the same symbol.
Yes, but any parameterized operator needs data for its representation, similar to how a closure is code + data.
Not sure if the fact that tensors have slightly more structure than mere arrays w.r.t. e.g. a change of basis is that useful, but it could be useful in proofs, especially if a FFT is involved somewhere.
Rearrange('b c h w -> b (c h w)')
Its kinda like how a matrix is "just" a rectangular 2D array of numbers in a particular basis.
- "Tensor computation (like NumPy) with strong GPU acceleration"
- "If you use NumPy, then you have used Tensors (a.k.a ndarray)."
Pure marketing drivel.