That is true but also people come to care about what they do. They may not be "passionate" about work but it is what they spend the most time doing, it is their main source of daily challenge and problem solving, it represents social interaction, and it is their main source of pride and accomplishment and personal striving. To just up and leave work leaves a huge void in their lives. People can learn to fill that void with art or gardening or various other things but it may take years and years of discipline and self-reflection to become satisfied.