Your last paragraph is just a list of reasons to ignore the game theory fact that your individual consumer choices won't make a difference either way. You say it is absurd, but there is simply no way of getting around that fact that the purely rational choice would be to shop at the place with the cheapest prices regardless of how you feel about the store you are getting those prices from, at least in terms of getting the best price and having an effect on which stores stay open. Sure, your arguments about meeting interesting people or having good ethics hygiene can be persuasive, but those are simply sidestepping the game theory rather than arguing against it.
I feel similarly, actually, and my gut reaction is revulsion to this fact, too. I hate the idea that my consumer choices actually don't have any affect on what type of business succeeds or fails. I want to believe that my choices matter to the world, but logical analysis clearly shows that it doesn't. We can't deny it, we can only ask, "ok then what do we do?"
My entire point is that we can't rely on the free market to fix these sorts of problems, because the free market relies on incentives, and those incentives won't solve problems like mistreating workers, polluting the environment, and any other externality of that sort. We can't simply hope that spending our dollars ethically will change anything, because it simply can't.
You might think I am immoral for pointing this out, but I feel like it is the opposite. The people who WANT us to believe that 'voting with our wallets' matters are the one's who want us to NOT do anything else. They know that our consumer choices aren't going to effect any real change, but they want us to think it can so that we don't do anything that could ACTUALLY make changes, like organizing politically. They want to promote the narrative that the free market is the way you change the world for the better, when it clearly isn't.
Honestly, this is the same as voting. People in power who want to maintain the status quo love to push the idea that you change the world by simply voting, but it isn't enough just to vote as an individual and hope it works out, for the same reason that an individual making a purchasing decision can't change things.
This is called the Paradox of Voting (https://en.wikipedia.org/wiki/Paradox_of_voting), because no individual vote actually matters.
We can only make change by organizing and coordinating action.