I just learned about the film in the recent book “Connecticut in the Movies” by Ileana Douglas. Burt Lancaster referred to The Swimmer as “Death of a Salesman in swim trunks”.
Just a guy who lost his job and couldn't replace it because of pride, cheated on his wife (maybe with multiple women?), and maybe became homeless?