“as_strided” and “sum” are all you need | Hacker News Reader

“as_strided” and “sum” are all you need

jott.live

76 pointsbrrrrrm10 comments

i wonder how this compares with einsum

> The einsum function is one of NumPy’s jewels. It can often outperform familiar array functions in terms of speed and memory efficiency, thanks to its expressive power and smart loops

it seems like einsum has been ported to torch & tensorflow

https://numpy.org/doc/stable/reference/generated/numpy.einsu... ; https://pytorch.org/docs/stable/generated/torch.einsum.html ; https://www.tensorflow.org/api_docs/python/tf/einsum

Great question! einsum is almost exactly broadcast + sum. as_strided lets you do “funky” strides to enable all the various convolutions (sliding windows aren’t easily represented with einsum).

Several years ago I used tensordot and from memory it seemed to perform slightly better than einsum. Not sure if that's still the case but I remember really enjoying using it.

Some short answers to this and following questions:

1. Einsum doesn’t AFAIK allow custom strides. So Einsum does only the mapreduce part (and that too only for multiplication & summation combo). To get a taste for the absolute brilliance of a slightly more generic interface, check out Tullio in Julia.

2. Tensordot etc use BLAS under the hood IIRC and can therefore parallelize automatically. The Einsum backend is good old fashioned single threaded C code, so it’s fast (and inflexible) but single core.

Einsum is a generalization of matrix multiplications to "tensor multiplications". It can take as_strided objects as input.

I don't think there exists a cuda-accelerated einsum, so the equivalent matmul is always faster? With cuda, it might be implemented by "exploding" the inputs to matrices, which means it can quickly run out of memory?

"...to implement the non-pointwise operations in a neural network."

> as_strided is trickier to explain. It allows users to reinterpet input tensors by specifying custom sizes, strides and offsets for the dimensions of the output.

This doesn't work as an explanation without defining what a "stride" is. You can't explain addition just by showing some examples of "before addition" and "after addition".

Here's what I found at https://ipython-books.github.io/46-using-stride-tricks-with-...:

> Strides describe, in any dimension, how many bytes we need to jump over in the data buffer to go from one item to the next.

So the stride in a 1-D array of float64 is 8 because you need to go 8 bytes forward to go from one item to the next. Change it to 32 and you have a way to access every fourth item without defining a new array.

thanks for adding this context! One caveat is that the PyTorch strides used in the post are defined by elements rather than byte-wise.

This is one of the best explanations I’ve seen of as_strided. Could have used this a few months ago when we had to use it to implement a CNN in class!

A neural network is not the useful default answer.

Try soap instead.

Reply on news.ycombinator.com