Calling someone who owns land and hires people to work it a farmer is like calling someone who owns a factory a "factory worker". In some technical sense maybe it's correct, but that's not what people mean when they use the term.
The word people use for what you describe is farmhand. It is the farmhand who works on a farm. The farmhand is the agricultural equivalent of a factory worker. Indeed, someone who owns a farm, but does not work on it, would not be considered a farmhand, but they most definitely would be a farmer.
It is technically possible for one to be both a farmer and a farmhand, but being a farmer does not imply that one is also a farmhand. They are distinct roles.