The people that download the book from these sites would never have bought them. Having your book downloaded by 10k people doesn't mean you've lost 10k sales, what it does mean is that you've got up to 10k people that wouldn't have bought the book anyway talking about it, effectively becoming word of mouth advertisers.
This isn't quiet as true for movies/tv series etc, because their value (entertainment time/price) is so much lower. Books on the other have usually cost $5-30 and will take 4-30 hours to read through. At that price point, very very few people will download the books to save what amounts to a single meal. Especially if you consider that so few people actually read several books per year (that's essentially $<30 per year "saved" via illegal downloads)
It could become an issue if a for-profit company could serve them legally however, I agree with that. It's hard to really talk about that, though ... It's a pure what-if/speculation after all