Named Tensors | Hacker News Reader

Named Tensors

nlp.seas.harvard.edu

70 pointstosh25 comments

This is a good but old article, and PyTorch has in fact integrated a lot of these suggestions over the past two years.

https://pytorch.org/docs/stable/named_tensor.html

Integration's still ongoing mind you, so it's not yet a complete replacement for 'traditional' indexing.

Similar functionality is available in Julia via [NamedDims.jl](https://github.com/invenia/NamedDims.jl/)

And in other packages, such as [AxisArrays.jl](https://github.com/JuliaArrays/AxisArrays.jl/) and [AxisKeys.jl](https://github.com/mcabbott/AxisKeys.jl). In fact, the AxisKeys README has [a nice overview of packages with similar functionality](https://github.com/mcabbott/AxisKeys.jl#elsewhere).

Right, NamedDims.jl is the one closest what the article describes. Very simple and light-weight, plays well with others.

There is a small zoo of packages attaching also labels along the indices, alla python's xarray. Perhaps it's a little too easy to write such things... but once we understand the design space the hope is to quietly take most of them to the woodshed.

I’ve been using Xarray recently (mentioned as an implementation of named tensors in the article) and I have to say I like it a lot. It’s really a game changer and makes my code a lot more readable.

Just wanted to give a major +1 to xarray. It is an excellent approach to labeled n-dimensional arrays, and has a good ecosystem/community in the Earth Science domains..

One nice integration with xarray is with an optimized data store like Zarr [1], TileDB [2] or Cloud-Optimized GeoTIFFs [3], where you can pull only the pieces of the nd-array relevant to your problem, rather than downloading a large monolithic netCDF4 file.

Another great integration in the xarray ecosystem is Dask [4], which is a very thoughtful abstraction over parallelizing computations across your nd-array.

[1] https://zarr.readthedocs.io/en/stable/

[2] https://tiledb.com/

[3] https://www.cogeo.org/

[4] https://dask.org/

I hate that this feature isn't stable yet. I actually ended up writing my own just to gurantee a stable interface.

The notion to put name derivation on operations is probably more trouble than it's worth. Random new ops will never support it, leading to never predictable results.

Still this is all better than pandas. <Shudder> pandas.

Pandas does give indexes and columns named entries, so this is less of an issue there, and everything has only 2 dimensions. In numpy it's a real problem. xarray is a library trying to fix that, though I haven't tried it yet. [1]

[1] http://xarray.pydata.org/en/stable/

I use xarray ever day. It's great for my workflows, which involve multidimensional datasets (typically scalar or vector fields evaluated or measured on 1D,2D,or 3D grids, sometimes including both Cartesian and polar or more exotic coordinates). I can read in an HDF5 file from a simulation & instantly make plots that automatically have meaningful axis labels. It saves me a ton of boilerplate.

There are some gotchas with the HDF5 reader trying to infer dimension relationships that aren't specified in files. Sometimes merging datasets can wrong: if a 'join' is supposed to take place along a dimension whose values are floats, the result will be empty even if there is machine-epsilon offset between the two versions of the same dimension. Or,you can get an 'outer product' by accident and crash the interpreter by trying to allocate too much memory.

> I actually ended up writing my own just to gurantee a stable interface.

Yeah, I do the same. Just write a thin wrapper and hook torch in the __init__.py.

But ragging on pandas isn't cool. Pandas is probably the most important factor driving python's popularity over the past decade.

Why is Python's popularity a good thing in itself? If all the people making plots in pandas just used R, the world would be very much the same.

Pandas promotes bad software practices in my experience with it's horrible bloated interface. It has cost me a lot of time-- for no benefit whatsoever. I am actually pretty sure it was released to slow the pace of competing quantitative hedge funds.

Discussed at the time: https://news.ycombinator.com/item?id=18823777

Is this how, in R, you can assign names to rows/columns?

Tensorflow's protobuf definitions actually supports naming dimensions: https://github.com/tensorflow/tensorflow/blob/master/tensorf...

Too bad nothing really uses or propagates those names though.

I’d love for something like this to be more standard. Writing complex stuff in PyTorch reminds me of writing assembly, except instead of constantly keeping track of state you have to constantly keep track of what the axes mean. Ideas like named tensors will have to become standard if “differentiable programming” is going to be more like regular programming.

"named dimensions" feels like a dynamically typed version of "giving each dimension index a distinct type", with the added ability to select nesting levels by name.

Is this not just making a map of strings to the integer of the dimension? It seems like this is something that would be done with enums or hash maps.

It would also change the broadcasting rules.

A tensor is a multilinear map that is invariant under coordinate change? What do those have to do with machine learning?

It basically means a multidimensional array in ML.

That's a severe abuse of terminology, almost to the point of ignorance. A tensor is an algebraic object (not limited to linear algebra) with a very specific definition.

What's going on here is a collision between the way certain terms are used in two different fields.

People who learn about tensors and vectors in the culture of Mechanical Engineering, Applied Mathematics, or Physics, are strenuously trained that a "vector" is not just any list of numbers, but a specific mathematical object with specific properties. (Similarly with tensors.) In that culture, the use of the term "vector" the way it is used in Software Engineering looks quite odd. An undergraduate might even have a problem set that shows various number triples and be asked to identify which ones are vectors and which ones aren't.

I was educated under that system, but spent years as a software engineer, so I am comfortable switching between the two contexts. When I wear my Software Engineer hat, I am quite comfortable with the generalized use of vector and tensor as done in software and ML, but for someone who has only grown up in one culture, it looks very wrong.

crudely,

    linear transformations : matrices :: multilinear transformations : tensors.

It has to behave in a certain way to be a tensor.

Given a vector space V over a field F, a tensor is a multilinear map

   T: V x V x V x ... x V^* x V^* x V^* -> F

You're mixing up geometric objects (multilinear transformations) with their coordinate representations. A geometric object, such as a multilinear map, is automatically invariant to the coordinate system. An array of numbers, like a matrix, is not.

Actually you're right. I was mixing up covariant and contravariant vectors with tensors. Although tensors do decompose into covariant and contravariant components, as your definition states.

Reply on news.ycombinator.com