The problem is that we need a way to say "actually; without exaggeration or inaccuracy" succinctly. Also, to say "I am using an idiom, but non-idiomatically."
"Literally" should be that tool. I understand that a word's meaning is, by definition, what people mean when they say it, but it is frustrating to have this communication tool drift away with nothing to replace it.
As another commenter said, this same thing happened to "very" and "really" in the past. So maybe it's time to let "literally" go and make up a new word for it?