But in any case I don't understand the claim of the article. If you can reverse the computation(say only use reversible matrix) you can do it for less energy?
if you're also batching matmuls isnt there an unavoidable information loss that happens when you evict the old data and drop in the new batch?
add(x, a) = (x + a, x - a), and add†(y, b) = ((y + b) / 2, (y - b) / 2)
It's a well known thing in thermodynamics that a reversible process doesn't increase entropy (heat). So in theory, a reversible computer consumes no power whatsoever, as it doesn't leak any. In practice, most power leakage in real life computers is due to wire resistance, not the irreversibility of computations. Also, even with a perfectly reversible CPU, you would need to expend some energy to (re)initialize the state of your computer (input) and to copy its results out (output). Alternatively, once a computation is done, you can always "uncompute" it to get back to the initial state without using any power, at the cost of time.
If you want an example of a real invertible computer, look into quantum computers, which are adiabatic (reversible) by necessity, in accordance with the laws of quantum physics.
* Actually, you can represent reversible gates with invertible matrices, and that has quite profound implications. A gate/operation is reversible if and only if its corresponding matrix is invertible. But let's not get into that here.
Let f: V -> V
g: V -> V x V is reversable form of f, where g(v) = (v, f(v))
g'((v, w)) = v
g' can be "fooled" with a fake w, but that is of no concern here. We trust our own chip!
f: V -> V
g: V x A -> V x A
with g(x, a) = g(f(x), b) for some value(s) of a. And b cannot be set to x, because then you can't find a back with g', and your function is not invertible.
In current chips we just charge and dump a bunch of parasitic capacitances every clock cycle.
In current computers, we’re nowhere near those limits anyway. Reversible computing is interesting research for the future.