There's nothing at all wrong with that! Just don't then make the leap to claim that this is a model for feeding the world.
My produce doesn't beat Walmart prices but then, we don't have the luxury of govt subsidies, scale, and undocumented underpaid labor. Once you start paying people what they deserve, and charging people what it actually costs, the prices are more truthful.
This type of food only seems like a luxury product because the scales are tipped so unfairly towards cheap/bad food.
Think carefully before you decide what is cheaper.
You're also assuming that local, small-scale farming has no impact. In fact to produce the same amount of calories requires vastly more land and more energy and the total impact is larger. Organic farmers also use pesticides and fertilizers; they just use different ones. It's not at all clear they result in less environmental damage. Is spraying a vast amount of nicitinoids over a larger area better than a smaller amount of organophosphates? Not at all clear.