Language is filled with those types of phrases, the one which bugs me once it was pointed out (even though I use it myself) is "to be honest...", which could carry the implication anything said without that qualifier may be dishonest. What including those phrases seem to come down to is an informal style, a bit more acceptable in a spoken conversation but for written it probably depends on the audience.

Something I'd wonder about is if usage of it has changed based on the medium people use over the years, whether that's in-person, telephone, writing letters, or computer/smartphone writing. Has using computers for short form conversations allowed conversational phrases to bleed into formal writing.