At least in my area, they aren't exactly known to be geniuses. The ones that have built up their business by buying out other farms and have tons of acres and employees? Sure. The usual "inherited the family farm" type? Not so much.
They have two small periods during the year where they're up before sunrise and stop working after sundown, assuming they live in an area with one harvest. Overall they work a hell of a lot less than most of us, and have more money. I have a friend that owns a business where he sells side-by-sides, four wheelers, snowmobiles, etc. The amount that many spend by buying all the newest toys every year is incredible.
The downside, of course, is that you need to be born in to it. A tiny 100 acre farm's land is worth north of 1 million dollars, not to mention a ton of other startup costs. Land isn't for sale very often, as usually at least one child wants to take it up. I don't even know how you'd approach starting from scratch unless you were already rich.
Note: I'm referring to crop farmers. Animals are hard work and I don't understand why any small time farmer still does it.