That seems like something of a defeatist argument, considering everything from the phone the person would be pulling out of their pocket to the pants the pocket is in to the shoes that are being tied to the door they're getting ready to walk out of was probably, in some form, produced by an unfair labor market or a company who has a less-than-stellar environmental record.

Removing a single consumer electronic device from your life isn't going to impact those global markets - they're problems best tackled at a different scale.