Was a horrible thing to say, but the context is rarely fully explained. Vidal called him a "nazi" first.
He served in the U.S. Army in WWII and was a member of Franklin Roosevelt's honor guard...so that probably was a pretty "deep cut" as an insult for someone with his military background.
Again doesn't make it right, but probably safe to say for him it was not the same as calling someone that from r/The_Donald in 2018.