Sometimes this is not straight forward for example when you have some responsibilities in your life, or when your career is too specialized with little to no transferable skill (like when you´re proficient at an internal process of a company).
Unless we could quit and be self sustainable without a job, then yes, at that point you could say one can quit anytime we want to find another place to work for.
Its not that 'finding a job is easy' its that we are not slaves and while we can not necessarily choose who we labor for, we can choose who we do not labor for.
You need to work for a living, and while you´re not a slave (because it´s not forced labor), your livelihood depends on it - and that has a conditioning effect on many levels (emotional, financial, relationships...)
Of course we always used a form of work to sustain ourselves, the difference might just be on the rewards we get from it.
Maybe that´s why some people exaggerate when they say they´re slaves of work, perhaps is a coping mechanism that allows them to frame that activity as something they´re being forced to do, rather than a deliberate choice they´re not happy about.