Stretching back further in the 20th century, Hoover lost due to the depression and LBJ declined to run for a second term because the Vietnam War was so unpopular they knew he wouldn't have a chance.
None of those are conventional political circumstances. So while you're right that it's about 3/4 incumbent races won, it seems that if there isn't a massive ongoing crisis, scandal, or political anomaly, the incumbent's chances are very good.