HPTT: A High-Performance Tensor Transposition C++ Library | Hacker News Reader

HPTT: A High-Performance Tensor Transposition C++ Library

arxiv.org

94 pointslainon26 comments

Why Ian this important? Not immediately clear to me.

Faster tensor transposition, which can speed up applications that do manipulation and calculation with tensors. Tensors are typically seen in physics.

For some reason, people use now the name "tensor" to refer to multidimensional arrays.

In the case of physics, the name tensor is well deserved and used with purpose---the object are mathematically speaking tensors (and where called tensors before computers existed). Their naming is unrelated to their physical representation in memory, even though they end up being arrays. The same is true for various engineering disciplines.

For ML, it just turns out that they can talk about their objects and the corresponding operations also mathematically as tensors.

Except that none of these operations are related to the properties that make a tensor a tensor, as far as I know. People were already operating happily on multidimensional arrays before.

"multidimensional array" doesn't quite roll off the tongue

They could have just been called arrays.

The reason probably is that tensors (in math/physics) can be represented as multi-indexed quantities, which are naturally expressed as multi-dimensional arrays in computers. 0-rank tensors are scalars, 1-rank tensors are vectors, 2-rank tensors can be represented as matrices (but strictly speaking are not matrices), and there are no specific names for higher-rank tensors.

Probably for same reason as they use the name "vector" to refer one dimensional array, "matrix" to refer 2 dimensional array. Tensor is generalization of matrix to more than 2 indices.

In physics people use tensor as a shorthand for tensor field.

A tensor is any quantity that transforms in a specific way under rotations. Essentially, it is required that a tensor describing some process gives the same answer no matter from which point (literally) one looks at it.

Tensor fields are spatial functions whose function value at each point is a tensor.

You might be familiar with more specific definition of tensors in physics where they come with full tensor calculus and transform in certain way.

More general mathematical definition of tensor is that tensors are multilinear maps from vector spaces to scalars. That's how TensorFlow and HPTT see these tensors.

> More general mathematical definition of tensor is that tensors are multilinear maps from vector spaces to scalars. That's how TensorFlow and HPTT see these tensors.

Oof, you have to squint very hard to see things that way.

Just because an image is physically a two-dimensional array of pixels doesn't suddenly make it a rank-2 multilinear map, and just because you have N planes of images doesn't mean you suddenly have a rank-3 tensor!

It seems to me that feature vectors in ML are used approximately like that, so I don't think it's so inappropriate. Take the preponderance of techniques like PCA, for example, features tend to be very often treated as rotation-invariant. Even if it's not 100% the case in the raw data, one very often wants to learn whatever features lie in subspaces that are invariant to linear transformations.

> Probably for same reason as they use the name "vector" to refer one dimensional array, "matrix" to refer 2 dimensional array.

Yes, that's where the rot started.

That "rot" likely started centuries before anyone even imagined a computer for an "array" to exist on.

Thanks. I've been meaning to read up on tensors as per tensorflow. Now I feel I don't need to.

Quietly raising my hand that this bugged me too.

Mathematics also uses tensor to refer to elements of a tensor product of two vector spaces. A tensor product of an n dimensional and an m dimensional space gives an n x m dimensional space. So to the extent that an array is a vector, a higher dimensional array is in a tensor product of vector spaces, and so we call it a tensor. Or, there's a correspondence between k-dimensional arrays an a tensor product of k vector spaces determined by a choice of bases of the vector spaces. And we're quite comfortable picking a "standard" basis for R^k, so we conflate the two ideas.

Now the part that always bugged me was that, at least in C++, a vector is an array that can change length, which is a very non-vectorial thing to do.

Mathematica has been using that terminology since 1988, and was written by mathematicians and physicists.

Apparently TensorFlow uses Eigen (another library for Tensors) and HPTT executes up to 27.4 times faster than Eigen.

Note that they only compared their solution with Eigen’s tensor transposition functionality. HPTT is not a drop-in replacement.

Does anyone know how much time a typical TensorFlow model spends in transposition routines?

No hard numbers to present, but it would be beneficial in long-sequence LSTM networks, because TensorFlow has to do time-major <=> batch-major transposition between steps.

Would be fantastic if this could be contributed to numpy.

Edit: source code is GPL licensed https://github.com/springer13/hptt

HPTT is intended as a drop-in replacement for any tensor transposition (see example on GitHub). Please let me know (best via email) if you run into any problems while integrating it.

cwise already does this and more, but it is written in javascript:

https://github.com/scijs/cwise

Reply on news.ycombinator.com