Two sparsities are better than one: Performance of sparse-sparse networks | Hacker News Reader

Two sparsities are better than one: Performance of sparse-sparse networks

arxiv.org

51 pointsAnon847 comments

Always cool to see research from Numenta. They don’t get as much love as DeepMind, Google Brain, and OpenAI because their results aren’t as flashy, but I do feel like they’ve got a principled approach to engineering intelligent systems distinct from that of the big players.

Sparse operations still don’t have good support in any of the generic SIMD instructions (avx2/512, Neon), or in GPU…

And memory access/cache should also be optimized for them to have even more power saving and better speed.

For ASICs/FPGAs implementations, however, there’s no such limit and perf gains are higher.

> Sparse operations still don’t have good support in any of the generic SIMD instructions (avx2/512, Neon), or in GPU…

This is incorrect.

For GPUs: Nvidia A100 (launched in 2020) has sparse matrix multiplication support for inference.

For CPUs: Check out SLIDE from 2019: https://arxiv.org/pdf/1903.03129.pdf

Sparsity could really be a good thing but so many times I've tried to use it and walked away disappointed in terms of accuracy.

It is already a good thing, but it currently requires a lot of engineering effort to actually get it to work with acceptable quality. It's not something that works out of the box like half precision, or for some models, int8. And to your point, for many production scenarios the ratio of engineering work vs performance gains is maybe not worth it. But for models that are going to handle massive load in inference it is worth it in my experience.

I expect that this will be made much easier in future with better hardware support and smarter sparsification libraries.

It's the power savings that are the real goal. If you can create and (re)train a network for incredible sparsity you can fit inference in some pretty low power envelopes. I think to that end the work required to get to adequate perform is justified

An abundance of sparsities.

Reply on news.ycombinator.com