They should worry. They were an industry that sold ornately decorated paper, and libraries were community or charitably organized warehouses of that paper kept available for people to examine for a time, then return. Now, like the music industry, the publishing industry are solely marketing and litigation organizations. Libraries don't even need books anymore, they can just become a quiet place to study that offers professionals who are experts at research who can help you find the digital things you want to study.

There is still a need for editors and digital typesetters, but lets not pretend that there is a huge need for publishers, i.e. the organizations that print and distribute paper books. There's still some need, and for people who want to earn an honest living, I'm still in the market for paper reference books. It's quicker for me to look up things in a paper book. I also like a nice annotated guide for an outdoor event or large facility. The preservation and digitization of books is an important job. It's necessary to have paper masters in case all the digital versions get deleted, or corrupted.

I'm also a little less hostile to copyright when referring to marks on paper, because it takes an investment to create the infrastructure to produce paper goods, which puts a publisher at risk. I'm honestly unconcerned and think society should also be unconcerned with protecting the investment you made to attract a particular author to your publishing house, and to convince people that author was worth reading. Whether that author publishes with you or another publisher, or self-publishes, is not something that society should concern itself with.

Scientific publishers aren't the only bad publisher. All publishers in the age of the internet are engaged in artificially recreating the difficulties previously inherent to the distribution of ideas, so they can remain the mediators and the controllers of the scarcity of those ideas. If human society is a large composite mind, transmitting information from node to node, they are Alzheimer's disease.

Anyway, publishers hated libraries before digital books, just like they hated used bookstores. All businesses hate competition. If a significant number of people started writing and publishing high-quality literature for free, they would organize to either destroy those works or find a way to tax or control their distribution, just like proprietary software and FOSS.