As it's an act of aggression, the people suffering may well demand the factory compensates them for the damage.
But actually, knowing that it will be held liable, the factory will spend more money on researching how to minimize pollution and will generally be placed in places where it does no damage to anyone.
It's all about aggression, really. The factory takes someone's rights unlawfully, it compensates for that. (And knowing it has to compensate, it'll avoid damaging in the first place.)
On contrary, a utilitarian government may decide that having some factory working is beneficial to the whole society so it's in "public interest" to let it infringe with people's rights to their property. So it says, "okay, it's bad; but it's in the public interest! we are going to allow this factory to work and it doesn't have to compensate you anything!"
Libertarianism says that even if having this factory running brings more "good" than "bad", it's still bad, because it aggresses against people's property. Property rights are above any "public good". Period.
(Oh, but if a group of armed criminals has control over some area of land, then it's probably okay to place mysterious "public good" above anything else.)