This way of thinking doesn't make sense. No one is claiming that you're responsible as in you need to fix this. We're claiming that you're responsible in the sense that you're part of the cause (or, in the language of Nagarjuna the "dominant condition" -http://www.class.uidaho.edu/ngier/307/nagarlec.htm) of the exploitation of people in third world countries. There are many other conditions besides you, of course.

The most important goal is to deeply realize your position and relation to other people - how you act after you realize this is up to you/your emotional makeup/your virtues/etc.

My claim is that because of the complicating factors (visibility and ineffectiveness) inherent in this situation you aren't able to fully understand this relationship.