I believe that both TeX (Knuth-Plass Line Breaking Algorithm) and InDesign (paragraph composer – expired US Patent 6,510,441) do this, so there is at least one open-source implementation that could be used as a starting point. Unfortunately though, the awareness for good and bad typography seems to be so low that this is probably not a priority for browser and e-reader vendors. It would be wonderful to have this as part of WeasyPrint or something similar.