1.5B parameters, only 300k neurons. The number of connections is roughly quadratic with the number of neurons.

I thought they had only applied the technique to 307,200 neurons. 1,000 / 307,200 = 0.33% is still low, but considering that not all neurons would be useful since they are initialized randomly, it's not too bad.