This is why I "soft retired" early. People haven't said it to my face, but I get the definite impression many think I'm crazy for "abandoning" work. I'm through hiking the PCT this year, something I never thought I would do until much older.
I'm not really a brave guy, I fear change, but you got to take your chances while you're still alive.