The State of Machine Learning Frameworks in 2019 (2019)
thegradient.pub
thegradient.pub
Something I'm super excited about for ML in Python is the variadic generics proposal[0]. It would allow us to statically check that the shapes of the input tensors to any operation are acceptable, and it would infer the shape of the output tensor as well. It might look something like:
x: Tensor[BatchSize, SeqLength, InpFeats]
weights: Tensor[InpFeats, OutFeats]
torch.matmul(x, weights) # acceptable
torch.matmul(weights, x) # not acceptable
Variadic generics should solve the problem that named tensors in pytorch[1] are addressing, except that (1) named tensors were not designed to be statically analyzed, so Variadic generics should speed up development speed considerablly, and (2) someone who wants to contribute to named tensors had to download the C++ source code and compile it (last I checked), which was a show-stopper for me. By contrast, the variadic generics proposal allows one to modify only the Python code, or even just maintain type stub files separately.
What's more, Pytorch is much more likely to adopt this so much quicker than Tensorflow, as the latter's enthusiasm for type annotations has historically been low [2]. Which is ironic, since TF forced the user to define the graph statically (in TF1 at least).
[0] https://www.python.org/dev/peps/pep-0646/