It could be that the islanders were supportive of the house being destroyed, or it could be that the islanders were scared to snitch on the bully that demolished it?
It doesn't need to be fear, necessarily. They live on an island of 120 people with whoever did it. Like the guy or hate him, they're gonna be seeing him every week from here til the end. Why turn on him for some bloke who doesn't even have a house here anymore?
Maybe everybody got their share. The materials had to go somewhere.