Note his forceful denials of an affair at the end, which we now know to be false. Of course, he couldn't possibly admit to such a thing back then, but it does make you doubt the rest of the account.
Do we? Do we know that an affair (as opposed to an unconsummated infatuation) existed at the time of the letter, as opposed to later? A quick googling indicates that Miss Ternan may or may not have returned his affections.
Despite what various public figures would have one believe, it is possible (if not terribly probable) to resist temptation…