The first plot of "Activation 0" appears to actually be the random input, if it corresponds to hs[0] in the code. The rest of the activation plots seem to all be strictly non-negative. The other plots with negative values are of gradients, not activations.