If they we’re just kept where they were you can always say they’ll eventually be scanned or have that potential.
I do believe the issue is a bit overblown but the core of it sounds reasonable to me.
Never mind that it's the 200th copy of some 1950's romance novel that nobody will ever care about, every time a library prunes its collection there is outrage.
Which is also not illegal and within some bounds and exceptions, a protected right across the globe.
If the books are rare enough there's a shared cultural value that is being destroyed.
Just because they are for sale doesn't mean that no one else would have bought them. Destroying them obviously destroys them, which obviously destroys culture. After all, if the books is destroyed, it will not be read.
Think of it like this. If I buy an island, and there are bunch of people who live on it, maybe they're day laborers or whatever, is it right for me to expel them, if I don't want them? Can't that be straight up genocide, if they have some culture indigenous to the island?
The same is true for books. If you destroy culture, you destroy culture. There is no magic which triggered "but I owned the physical book" which means you didn't.
It doesn't matter what property relationship you have to a thing: whatever property relationship you may have to it, you still do what you do, i.e. you take whatever action with respect to it as you take with respect to it.
Another good example is food. If there's a shortage of it, and you still have a great deal, you may think "eh, what does it matter if I accidentally burn some, I save time through my carelessness" but if there are others who aren't getting any, you may be killing people, and may be despised for how you use "your property". The same is true here. Why should one not despise one who deprives others of rare texts, that may even be lost, and thus cause cultural destruction of some culture that may actually be rare, by treating the carelessly. That rare book on model building or whittling from the 70s may actually be important.
You scan books though and all the value they provide is still fully available in bit form, because books merely contain information. And of course being in bit form means they can be trivially copied, technically. But there's also the legality, which determines how many copies it's legally permissible to retain. The latter becomes the bottleneck for your preservation of "culture" because said culture could be made trivially available to anyone with an internet-connected device, if only rights holders weren't screaming foul (remember what happened with Internet Archive during/after Covid?[0]).
Ultimately there's no destruction involved when books are scanned, just a change of format which helps to ease management and for legal compliance.
[0] https://www.libraryjournal.com/story/internet-archive-loses-...
You say "culture", but is is culture, and culture that was interesting enough to write down. Writing a book in the 70s isn't a trivial thing, there are gatekeepers, editors, publishers etc., even for your little book about local history, and the information is valuable to the community it's about. It affects their internal cultural transmission.
You're ignoring that they're also scanning them. Which means that we're one copyright law change + some minor incentives away from making these archives actually useful, vs. the book silently getting destroyed in the bookshop's dumpster because nobody bought it.