I want to emphasize the last bit though about “building an empire”. It depends on what your metric is. You can be rich and be broke. You can be poor but wealthy. The empire here would be in community and not in money. Giving back by selling your produce, locally, and helping the community. That, is satisfaction. On top of the satisfaction you have from growing your own food of course.
I go to a farmers' market in the Bay Area. It draws farmers from a 100-150 mile radius. Being California, you get produce from a lot of different climates; some things need a cool coastal climate, some need the heat of the central valley. You also see the season of crops move. Things usually start in the south where it's a bit warmer, and the season (and vendors carrying) that crop moves north over the next few months.
The community approach can really only get you some amount of diversity in produce that does well in your area, and maybe a bit from the soil type. Talk to anyone who's grown squash will tell you there's a month where their one plant has more squash than they know what to do with. The problem is everyone in the community has the same squash surplus at the same time.
I appreciate the idea, I just think that produce production will always tent towards hobbyists and professionals because of economies of scale.