I don't think there is much difference between those and big brands. Same with clothes. There is no difference between a super low cost brand, H&M and Versace. Or actually there is, the designer brands have often proven to be manufactured in even worse conditions.
They will suffer more without the job. You think not buying from them will help them...how?
Buying fair trade supports an alternative over the status quo.