It's the functional equivalent of someone buying the book and then passing it on to the next person who wants to read it.
With pirating, perhaps one person bought the book originally and then it gets sent to an infinite number of people who permanently have it in their collection. The author gets paid practically nothing.
The short answer is that libraries buy books, pirates don't.
In regards to physical books, a good place to start is understanding "first sale doctrine"[1], which allows both libraries and you to do what you want with a book once you're purchased it.
First sale doctrine does not apply to DRM-encumbered ebooks, so libraries must buy as many licenses as they wish to loan, paying three-to-five times the retail price for each limited-time license, and re-purchasing those licenses when they expire (typically after two years).
[1] https://www.nolo.com/legal-encyclopedia/the-first-sale-doctr...
Here is an interesting answer on the topic.
https://www.quora.com/How-does-a-library-pay-royalties-to-th...
They also provide a net benefit to society by encouraging knowledge and literacy. There is a social expectation that physical goods are sold in most cases without expectation that seller will retain contractual control in order to derive maximum profit. Example nobody liked when Keurig sold coffee pots that wouldn't work with generic or indeed even older official pods via electronic tags in pods.
Also keep in mind that that we all exist in a society there is no reasonable expectation that you have a moral right to be able to use societies apparatus to maximize profit if its at societies expense. Limits are the norm.
Some of us see the virtue in authors getting incentivized in theory but think copyright at least as practiced is a net negative for society because it stops the free spread of information that would otherwise better enlighten the world. This way of thinking actually dates back to some of the founding fathers.
Nobody "deserves" to be paid because someone has arranged a pattern of bits in a way that they "own". Different laws have different up sides and down sides and we ought to pick the set of rules that results in the highest benefit/lowest cost to society. This unlike ownership of imaginary property has moral force. By choosing to treat the current rules as given good you have missed out on the opportunity to make a useful argument about the relative utility of different strategies.
Arguably the current dynamic where piracy is technically easy but practically discouraged might be far more optimum than one in which copyright was actually maximally enforced because the people who have plenty of money value convenience and pay out at a substantive portion of what they would pay in a maximum enforcement scenario whereas those who would otherwise go without are able to.
On net you end up with multiple times the positive effect of a maximum enforcement scenario while still funding authors having a decent life.
For myself I think artificial scarcity of any variety is an attempt to preserve a business model based on actual scarcity that doesn't make much sense in modern context. Ultimately it wasn't the VCR inventors job to justify to the copyright industry that home video industry made sense. The logical step towards phasing out copyright would be limiting it to a sane time frame like 7 years wherein most of the money is actually made in the first place.