I agree nostalgia can cloud judgement but there has been a fundamental shift in the nature of farming in the last few decades.
Btw, this is a perspective from Europe - the same thing has happened with family farming here. While I didn't grow up on one, I spent a lot of time on my grandparent's farm.
Farming has never been easy but the amount of capital required (represented by land, livestock and machinery) to run an economically viable farm has increased enormously increasing the risk. By viable I mean profitable enough to provide an average family with a median income.
My grandparents raised 4 kids, putting 3 of them through 3rd level education and never really wanted for much purely on income from a 90 acre farm. All their neighbours and the entire landscape was covered with family farms of this size plus or minus. What's left now are some old bachelors barely surviving or part-timers (who generally inherited the land) with "real" jobs who keep some dry stock or rent out their land to big producers.
It's sad to see a way of life disappear but I don't see any alternative. From an economic perspective, it can only be supported by transferring income from others to small farmers. This is effectively what happened in the past through higher prices paid for agricultural produce. As a share of disposable income, spending on food is a fraction (well under 10% in the west) of what it was in the past - which hurt the poor disproportionally.