The heroes in this game are the instructors who are creating open-source textbooks.
I think you are seriously overestimating the financial gains such a professor would make. I've taken several classes with profs who teach from their own textbook; the profit per book sold that a prof receives is usually minimal (either because the book is sold for near cost or because the publisher takes most of the profit). In fact, most such textbooks (essentially just rigorous lecture notes) are downloadable for free anyway.
If you want to make extra money as a professor, writing a textbook and then forcing it on a single class of students is about the least effective money-making scheme I can think of.
Later on I took MITx's prototype 802.11x online course, where I learned far more and managed an A. In that case the professor actually cowrote the textbook, but there was a free online copy for the online class, and the textbook was actually good.
Edit: in another case, another course at the same university used a custom printed textbook sold exclusively by the university for well over $100, that was just a few chapters from one $15 book and the complete contents of another $20 book bound together.
I know that there were ulterior motives on the professor's part because I specifically asked to just print it out myself and was told that the TAs had been specifically banned from allowing students access to the PDF version for printing.
This was a 300 person lecture taught twice per year.
300$352 is $21,000. Suddenly that looks pretty appealing to the professor who thinks he could use a new bass boat this year.
As to his profit, I'm not sure how much he made off of each book, but the book is used across the world. The only reason I think any instructor would use his book is because it's a niche subject matter; the mechanics and theory of hydronic heating systems.
If Dr. Hydronics ever reads this--you're probally a nice guy, but you made a relatively simple subject complex. Funnel your sentences. Use numbers. Don't use unnessary words. KIS(keep it simple).
It all depends on the prof and the content.
My wife bought a "textbook" for a language class and it was, and I kid you not, 300 loose pages with binder holes punched in them. See, it's a textbook AND a workbook! And sure, some of the pages were workbook pages. But there's no way any bookstore would buy that back as soon as the shrinkwrap was broken.
There is no excuse, though, for having to sign up for an account with a commercial publisher for exercises that are a part of your grade in that class. Normally you would pay a bunch of TAs to administer and grade the tests, but the university is too short of funds for that. So they outsource the grading to a commercial entity.
It was hand-bound by some poor soul in the printing office and was clearly churned out with a laser printer, but it got the job done and saved everyone a bundle.
If the prof made some money off of this, I'm fine with that.
Of course, they will still suggest looking at some textbooks, but only on the rare occasion that they don’t want to include a proof or more information in the notes themselves - and then these books are naturally available in the library’s reference holdings.
I would definitely drop a class where a professor started a semester by doing something so ethically gray.
2. There is really no way to force a student to do anything.
Disclosure: I list some printed texts and reference books as required on my syllabi. I do this because I give open-book, open-note exams; but I do not allow the use of electronic devices (except pocket calculators).
In another case, our professor contacted the author of the book and got cheap locally printed and bound copies. Something like $15 vs. $180.
Most of the professors gave away the slides of the course, and even more additional material; these ones gave nothing but a bunch of literally useless tables and charts, completely worthless for the exams.
Needless to say, this textbook was the only and obligatory literature for the course.
Where I studied (undergrad, at least), there were conflict of interest rules -- professors were not allowed to collect royalties on books they use in their classes.