I remember people saying that, but I had always understood the primary benefit of cloth and other masks was to limit the wearer's ability to spread virus-containing droplets if they were infected by asymptomatic (or symptomatic and very reckless, I suppose). I thought it was widely acknowledged that they provided very limited protection for the wearer. "My mask protects you, your mask protects me." This is why, for example, there were recommendations to not wear vented N95 masks.
Of course, the fact that the mask doesn't form a tight seal around the mouth surely undermines its ability to provide both forms of protection, to some degree. But I seem to recall a fair amount of discussion of the fact that masks could still be fairly effective at reducing the number of virus-containing droplets expelled into the surrounding environment even in the absence of a tight fit. (And, for what its worth, that has always made fairly intuitive sense to me. Though I acknowledge that's not a very good reason to believe it.)
Maybe others' experiences differed, but I thought this was very clearly explained by the 'authorities.' Though that didn't stop people from misunderstanding or, in some cases, spreading misinformation to score political points by making mask proponents look stupid.