Really? I know professors who wrote textbooks in their spare time. Universities could give professors a paid year off (no teaching responsibilities) to work on textbooks. Time is a non-issue here.
"...or skills to create diagrams..."
Professors, who spend their time writing research papers that are usually filled with diagrams, don't have the skills to do so? That's an interesting assertion. There are plenty of software tools available for creating diagrams.
"handle formatting and typesetting issues"
Hm, I think there is a tool for that one too: LaTeX. What, it's too much to expect a professor to know how to use LaTeX? Well, there are several other tools that could be used. These problems were more or less solved decades ago. Professors routinely format and typeset their articles; formatting a book is not substantially different.
"write hundreds of homework problems"
I know professors who do this: they write homework and test problems for their courses.
Really, I think you underestimate what an experienced professor could do in a year. Even if what you were saying was true, it would not be hard to solve: universities could hire experts on formatting and publishing to work in their libraries and assist professors. It is also not necessarily the case that every professor would have to write whole books; if books were being shared by universities via their libraries, professors could make corrections and improvements to books as needed, with the changes being propagated in a manner similar to patches in open source projects. We have all the technology that is needed for this, we only need to put in the effort to organize such a system (but I won't hold my breath).
At whose expense? Taxpayers and the students themselves, of course. There is no such thing as a free lunch.
On top of that, there is no sense going against the specialization of labor. Better leave the professionals (in this case, the publishing companies) to do their thing.
As for the "professionals" at the publishing companies, what exactly is their profession? We do not need publishers to copy books for us anyone, because we have computers that can copy the books faster, cheaper, and on a much greater scale and if students need printers, they have access to them. So what professionals would you be referring to? Textbook publishing companies are not academies where experts in a subject spend their days writing about that subject; universities are the academies where experts (i.e. professors and researchers) are paid to write about their area of expertise. Textbook publishing companies hire people who can make fancy covers (which adds no value to books), double check typesetting (which is a redundant task, since librarians and professors can do that), and operate machines that print and bind books (but we do not need that anymore, because we have automated ways to distribute books). So in what way are we "going against the specialization of labor" by asking for books that are CC-SA licensed, written by professors, and shared over the Internet?
When diagrams and typesetting are needed though, this is the perfect task for the students to attend to. I'm seeing this right now in the free online classes at Coursera and Udacity - teachers publish notes in lectures and students are providing transcriptions, translations, diagrams, deeper explanations, and overall creating massive improvement on the originals, in many cases creating an online textbook in pdf or wiki format.