That's kind of exactly what floats are. You store the log2 of the number, a bit for its sign, and in what remaining bits you have left some fixed-point scaling between adjacent powers.
I think they mean storing just an exponent instead of an exponent + mantissa. Makes multiplication much easier to implement.
No, that is not what floats are. A logarithmic number system literally just stores the logarithm of a number (and a sign bit) and manipulates it as a logarithm. The significand is 1, always. So multiplication & division are simply addition & subtraction, respectively. But this simplification for multiplication, division, roots, and powers is counterbalanced by more complex addition and subtraction.[1]
how do you do FMAs then? You're making one operation cheaper, but making the other one expensive.
“You’re making one operation cheaper, but making the other one more expensive” is the wrong way around. Say it this way: “You’re making one operation more expensive, but the other one cheaper.”