The breakeven point seems to be around 50-100 uses depending on the exact kind of reusable bag, so you’re probably winning: https://ourworldindata.org/grapher/grocery-bag-environmental...
Okay, let's stop burning coal then, because we use electricity for a lot more things than just recycling.
Similarly, if cotton is that bad for the environment, maybe we need to fix that generally rather than just switch to disposable plastic bags for this one minor use case?