To be fair, we wouldn't have "literally" without a metaphoric interpretation of "literalis," "of or relating to letters."

I always find this particular example to be somewhat self-contradicting for this reason.