I actually don't think they had it bad. Sure, they worked very hard, but that was their life. The good part is that they were fully self-reliant. They had food, were able to protect themselves, were able to fix anything that was broken, they had a strong community with the other farmers in the neighborhood. In rough times, I think this is very valuable.
I'm sure they didn't want to become knights and die in some war. Just because the scholars looked down on them, didn't mean they looked down on themselves, or were all stupid. Because back then you didn't have as much chances to study, I'm sure the intelligence of farmers was very diverse. About 50% of my cousins have a university degree, and most of the rest bachelor. So our genes can't be that stupid ;).