An EPUB is just a ZIP archive of XML and XHTML files (plus other assets, like images). Partly, I suspect, because of the dearth of actively maintained open source projects in the space, and partly because of the nature of tech in the book publishing industry, EPUB generation software used by authors and publishers often messes up this spec, which means that EPUB readers sometimes need to have fairly complex fallback logic for trying to figure out how to render a book. Also, because EPUBs are ZIP archives, some readers may either unzip the entire book into memory or "explode" it into an unzipped directory on disk, both of which may result in some slowness, especially if the book has lots of large resources. The newest Brandon Sanderson novel, for example, is ~300MB _zipped_.
Additionally, and perhaps more importantly, EPUBs (and I believe MOBIs as well) represent content as XHTML and CSS, which means that readers very often need to use a browser or webview to actually render the book. Precisely how they deliver this content into the webview can have a huge impact on performance; most browser don't love to be told to format entire novels worth of content into text columns, for example.
Terry Pratchett books are notorious for that. Some tools EPUB authoring tools artificially introduce breaks, but you can't rely on them.
So it's possible :)
I'm now actually working on a Calibre-Web change to strip the audio and media overlay from the books it serves via OPDS.
Then I'll need to tackle cross-device progress sync. This turned out to be surprisingly tricky.
So I'm trying to design a progress sync protocol. My current idea is to just use several words from the text itself to unambiguously pinpoint the position within a section (chapter).