I've done pretty much this for the few e-books I've bought. None of the e-book applications for OS X do proper justification with automatic hyphenation, and some of them (such as B&N's eReader) don't even use subpixel antialiasing. The DRM is generally pretty weak, and it only takes a few regexes to convert them in to valid LaTeX code. After you clean it up a bit and add the semantic markup necessary to get proper chapter divisions, etc., you get a PDF that's a true book, suitable for printing, with typography that is actually readable.
I've only tried with this two samples. The results that LaTeX gives always a feeling a being a lot better than all other solutions/pdf generators I've seen.
PS: Don't say the DRM is too weak in public ;)
I just don't like the fonts in LaTeX. From what I remember, the Memoir class dosen't work to well with XeTeX.
In my experience (admittedly only with recent versions) there aren't any problems that crop up when using XeTeX with the memoir class for simple stuff like ebooks. The only real downside is that the microtype support in XeTeX isn't as good as in pdfTeX, but LuaTeX is pretty close to providing the best of both worlds.
You can \usepackage{palatino} for Palatino, other postscript fonts are quite accessible.