A lot of it is about animals for me as well, because no mass-produced food is really clean of animal suffering.
Even if you buy organic soybeans, if they're coming from the supermarket, and you're paying a dollar for the bag, that dollar is sending a clear and unmistakable signal to the supply chain: "please replace this bag of soybeans".
From there, a whole chain of events unfolds... a bag is pulled from the case, a case is pulled from the pallet, a pallet is pulled from the warehouse (on a diesel truck), and so on. There's lots of diesel and machinery involved, and all the way at the other side, there is land being cleared from animal use, probably with some animals dying as a result, in order to grow some more soybeans.
Sure, a good portion of this is unavoidable. Our entire existence has a baseline of suffering which we cannot get rid of completely. But if I can reduce my contribution to it when it's totally unnecessary (because there are 50 bags of organic soybeans in the dumpster, for example, or there are are many soybean servings left over at the cafeteria after it closes) then I'd rather do that.