Quoting a character in one of Stanisław Lem's stories, "People do not really want to live forever -- they simply do not want to die." Always struck me as a rather profound take on the whole immortality deal.
Von Neumann definitely didn't want to die. According to Wigner: "When Jancsi realized that he was incurably ill, his logic made him conclude that he would cease to exist, cease to have thoughts. The full content of this was incomprehensible to him and it horrified him. We all know this and try to accept it, but it is always difficult. It was heartbreaking to watch the frustration in his mind when the hope was gone. His fate was unavoidable to him, but it was still unacceptable."
His friends and colleagues were very disappointed that at the very end he turned to religion to help him cope with his fears. He even had a priest visit him for last rites. (Source: A Beautiful Mind)
To add another source, Greg Evan's Permutation City is what drove home for me how profoundly inhuman immortality is.