I was thinking about this idea (using only additions for inference) before too. The article says about using 1-bit values for weights, but there is also another option: use integers for weights, but allow the output of a neuron take only 0 and 1 (for example, 1 if output is > 0 and 0 otherwise). In this case we do not need multiplications as well, but cannot save memory for storing weights. Maybe it has some other advantages, who knows. Never had time to research this further.