When you buy a print book, you are buying a physical license token for a copyrighted work, bundled with a single copy of that work. The license is attached to the copy.
These are actually separable. If you remove the cover from a book, you are destroying the license token, without necessarily destroying the [now unlicensed] copy. This is something various entities in the book industry may do, to refund unsold copies from the distributor without incurring return shipping, warehousing, or handling charges. Instead of sending it back, the book is rendered unsellable by removing the cover with the ISBN on it, the paper is tossed in the trash, and nobody gets paid for the unsold copy. You can theoretically fish the pages out of the trash and read the book, but that's now technically piracy, because the copy is unlicensed.
If you rip the cover off of a book you bought yourself, no big deal. You still have the license, because you bought it, but then you can't resell your copy, because the license is no longer embodied in a physical token attached to the copy. You can't prove you have the license, and any potential buyer couldn't be certain that you transferred it to them.
For an audio-book on physical media, when you buy it, you buy the license implicitly attached to the copy. But since copyright is a mess, a recording of audio is a different copyright from the material used to generate the audio. An audio-book of Jeremy Irons reading a story has a different copyright as an audio-book of Bobcat Goldthwait reading the same story. You're not licensing the story itself, but a specific recording of the story. It's the same reason why a piece of music can be in the public domain, while a recording of it made by the FooBar Ensemble in 1991 can still be copyrighted. That theory of copyright is also how museums and collections try to claim copyright on digitizations of visual art that has long been public domain. There are a lot of tests about "sufficiently transformative" and "slavish copy" and whatnot. There's also a lot of money in ensuring that cross-media copyrights remain separated.
From the publisher perspective, someone has a license for an audio recording. They probably also sell licenses for print copies, and licenses for e-books. And these are all sold individually, not as a bundle, because they are allowed by law to treat all these as separate works.
From the consumer perspective, buying the audio recording is implicitly licensing both the specific recording and the work on which it was based, because otherwise, how could one possibly legally enjoy the content of that recording? So the consumer considers it perfectly acceptable to transform the format of what they believe they have licensed. I paid; this copy is mine now; I can do with it as I please. Parent post reinforces this. "I'm paying for the author's story + actor." But you're not. You licensed a copy of a recording, period. It's a derivative work, and you're piggybacking on the recording-owner's licensing of the original story, that gives them the right to make an audio-book.
This confusion is not the consumer's fault. It arises from the non-intuitive cross-media parts of copyright law. It arises from the publishers and distributors of copyrighted works exploiting the ambiguity between buying a license and buying a copy. They would dearly like to have their cake, and eat it too. When people buy cakes, and then eat them (as one does with cakes) they get upset, because they sincerely believe that selling a cake and selling the right to eat a cake are completely different things.
If they choose the non-transferable individual licensing model, they can no longer sell the same work to the same person multiple times. If I lose my file, I still have the license, and I can either demand a fresh download from the distributor or acquire one from another source without being considered a pirate. If they choose the license-attached-to-copy model, they can no longer capture the resale market. I can read or listen to my book and then resell it to someone else. What they want, but should not have, is a revocable non-transferable license attached to a single copy, so that if I buy a copy, and lose it, I still have to buy another copy, but if I then locate my original copy again, I can't resell it or get a refund for it. That model is worth more money to them, in their estimation.
In my opinion, the publisher argument is bullshit. But it is founded in law. The law happens to be an ass. So they will likely win, until a federal appellate judge decides that they want their public service career to be destroyed by Disney et al.