It's unfair because they are only enforcing the law as written, and it's written by representatives of the people of that country(1), and thus they are only doing the will of the people. If you don't like it, be angry at the people responsible for the law, not the agency.

It's like hating the policeman because they keep busting you for drugs. It's not his fault they're illegal. In fact, many policemen feel the drug laws are stupid. It's equally likely that many in the INS feel the law is unjust. What are they going to do, though? Break it?

Responsibility flows upwards in such matters, especially where governmental mandate is concerned. Blaming officers of the government for merely enforcing the will of the people is misguided.

(1) Doesn't count for non-democracies, obviously, but even in those cases they'd just be enforcing the law of the land, with little individual perogative.