Really? I know professors who wrote textbooks in their spare time. Universities could give professors a paid year off (no teaching responsibilities) to work on textbooks. Time is a non-issue here.
"...or skills to create diagrams..."
Professors, who spend their time writing research papers that are usually filled with diagrams, don't have the skills to do so? That's an interesting assertion. There are plenty of software tools available for creating diagrams.
"handle formatting and typesetting issues"
Hm, I think there is a tool for that one too: LaTeX. What, it's too much to expect a professor to know how to use LaTeX? Well, there are several other tools that could be used. These problems were more or less solved decades ago. Professors routinely format and typeset their articles; formatting a book is not substantially different.
"write hundreds of homework problems"
I know professors who do this: they write homework and test problems for their courses.
Really, I think you underestimate what an experienced professor could do in a year. Even if what you were saying was true, it would not be hard to solve: universities could hire experts on formatting and publishing to work in their libraries and assist professors. It is also not necessarily the case that every professor would have to write whole books; if books were being shared by universities via their libraries, professors could make corrections and improvements to books as needed, with the changes being propagated in a manner similar to patches in open source projects. We have all the technology that is needed for this, we only need to put in the effort to organize such a system (but I won't hold my breath).
At whose expense? Taxpayers and the students themselves, of course. There is no such thing as a free lunch.
On top of that, there is no sense going against the specialization of labor. Better leave the professionals (in this case, the publishing companies) to do their thing.
As for the "professionals" at the publishing companies, what exactly is their profession? We do not need publishers to copy books for us anyone, because we have computers that can copy the books faster, cheaper, and on a much greater scale and if students need printers, they have access to them. So what professionals would you be referring to? Textbook publishing companies are not academies where experts in a subject spend their days writing about that subject; universities are the academies where experts (i.e. professors and researchers) are paid to write about their area of expertise. Textbook publishing companies hire people who can make fancy covers (which adds no value to books), double check typesetting (which is a redundant task, since librarians and professors can do that), and operate machines that print and bind books (but we do not need that anymore, because we have automated ways to distribute books). So in what way are we "going against the specialization of labor" by asking for books that are CC-SA licensed, written by professors, and shared over the Internet?
When diagrams and typesetting are needed though, this is the perfect task for the students to attend to. I'm seeing this right now in the free online classes at Coursera and Udacity - teachers publish notes in lectures and students are providing transcriptions, translations, diagrams, deeper explanations, and overall creating massive improvement on the originals, in many cases creating an online textbook in pdf or wiki format.
Instead of required text books they have lecture notes for most of the classes that are provided free on the website that might as well be a textbook considering how thorough they are (eg: [2]). One or two intro classes had a reader version of the lecture notes, but that was well over 1000 pages, and again, not required since it was provided free on the course website.
For the few classes that did have required textbooks, most of them were books that had been out for at least 3-5 years, so used copies were easily available (eg sicp, K&R C, Vazirani's algorithms, etc.). For the very few classes that had textbooks with a recent version the profs mentioned in the first lecture that the old versions would work fine, but that the students would have to figure out where the assigned reading was if the chapters didn't match up exactly.
[1]: "There is no required textbook for this course, since none is altogether satisfactory, and all tend to be ridiculously expensive. Officially, we'll rely on notes that I provide (on paper and on the web), but many people feel more comfortable if they can supplement this information with other treatments. You might want to take a look at the following books:
Compilers: Principles, Techniques, and Tools by Aho, Sethi, and Ullman—also known as "The Red Dragon Book". You won't need the latest edition.
At least one student recommends Thinking in C++, by Bruce Eckel, which you can download by clicking here.
Python Essential Reference (3rd edition) by David M. Beazley. This is also on-line (see the bar on the left) and at bookstores, on-line and otherwise." - http://inst.eecs.berkeley.edu/~cs164/fa12/info.html
[2]: http://inst.eecs.berkeley.edu/~cs164/fa12/notes/notes.pdf