> It can be optimized away by a compiler to just fixed integers, sometimes

Yep, SBCL does range analysis and can do inline fixnum or unboxed arithmetic.