Portable EPUBs
willcrichton.net
willcrichton.net
1. absolutely we should have a single-file, portable ebook format, and since PDF doesn't reflow text then it's not that.
2. HTML + CSS in 2024 is capable of reproducing virtually any kind of printed medium, but it can also reflow text.
3. I don't personally think JS should be a requirement, but a lot of conversations break down at this point so let do my best to explain and please understand I'm not simply being religious about this. In my view, an ebook is a book that can change its size in a way that makes sense. Bearing this in mind, I believe that if a reader has JS turned off, this book should work as intended. In other words JS shouldn't be required for it to perform its basic function. However, if there is some necessity to add interactivity or to augment the book, then yeah, why not, use JS. That's what it's there for. However, from the perspective of being a book, if it doesn't work as a book without JS then it's a bug in my view. (for standard ereader capabilities, I think the browser should offer that, but the book itself shouldn't be shipped with an ereader)
4. I think it's a mistake to embed all the styling as that may violate some people's CSP's. It's safer to specify separate styling as resources relative to the HTML, and ebooks should forbid loading resources from a separate domain. In this way, ebooks would always work offline, but it has the added benefit of working online too and would automatically adhere to the strictest possible CSP. This achieves the same goal as offline, but in a safer and more universally compliant way.
5. Finally, just distribute it in a zip file. That's how ebooks already work right?
What about self-extracting ZIP files like this page? https://gildas-lormeau.github.io/ (note that it includes the CSP to make it safe)
In any case, ideally, any ebook solution should have all resources loaded as files relative to the current document, and nothing inline. Like this, the book would be compatible both offline and to be hosted online, without requiring any changes to the book.
The one file thing is cool, but again it requires JS to show anything, so that's not really inline with what I was talking about. In my view, an HTML document should work like a book, and any JS is purely to augment and extend that book if necessary. Those situations are rare though, and most JS is just to give a reader like experience.
I think if someone is going to go through the trouble to host a book, they probably don't mind unziping a file before putting it on their server. The one file thing I said earlier was more about sharing the book, similar to an app package. but once it's on someone's servers presumably it's ready to be read.
Basically .webarchive
The JS is not essential, there's nothing to stop you treating the file as a ZIP file and unzipping it beforehand to view it.
https://github.com/gildas-lormeau/SingleFile/blob/master/faq...
Sort of, in the sense that any unzip tool will unpack an EPUB file (possibly after renaming it to a .zip extension rather than .epub).
However, it doesn't necessarily work the other way. You can't just zip the files at random and wind up with a valid EPUB, even if that exact same set of files was a valid EPUB before it was unzipped.
The "mimetype" file in an EPUB zip is special. It has to be the very first entry in the archive. It also has to contain the string "application/epub+zip" (and nothing else) and must be stored uncompressed.
A surprising number of zip tools make it hard to do this (e.g., by altering the file order as more files get added to the zip, or making it hard or impossible to store one file uncompressed while compressing the others). Most of the command line zip programs can do this with the proper command line flags, but zip libraries often make it a PITA.
Source: have written EPUB generation software.
One thing I missed about working with old .doc files instead of .docx, was that it was very fast to search a folder with hundreds or thousands of files for a specific word. Not possible for zip formats. (I just saved a file in .doc right now and open it with a hex editor and it does contain the contents in plain-text.) It is a problem to search zip formats or pdf files with grep, and I really doubt that the zipping is necessary for text with maybe a few thousand characters.
Though pattern matching in files feels very fragile - simple text patterns would still rely on the text being kept in a contiguous chunk and no embedded data/markup within the section you're searching for...
Though I generally see the prevalence of zip files a result of people assuming that collections of files/streams need to be packed into a single object for "user simplicity", but we already have this with OS support in the form of directories. It just seems people have embedded the assumption that tools and UI handle directories as single units poorly, when that doesn't really need to be the case.
Fundamentally, directories are problematic because they are user-facing. When used as a file format, you risk running into situations where the user has (sometimes inadvertently) deleted some of the files or edited them, partial copies etc. Making it all into a single zip file makes it clear that it really should be treated as a single atomic unit by any casual user, while still allowing power users to hack on it when needed.
There are zip-aware grep tools that you could investigate if they fit your workflows.
There's also nothing stopping you from storing the contents of zips as unzipped directories and rezip when/as necessary. (I wrote a tool called musdex years ago to automate that flow in the context of source control: source control the contents of something like .docx expanded into a full directory structure, but preserve the ability to "double click the Word file to edit".)
As a wrapper of "collections of deeply related files, some which may be text and some may be binary", ZIP is one of the better choices that we have (compare to TAR or MIME envelopes, for instance).
True, ZIPs are hard to grep. However, gzip and progeny come bundled with convenience scripts as zgrep, bzgrep, xzgrep, etc.
Maybe use a grep-friendly format if that's important to you, or otherwise a filesystem with transparent compression (btrfs, ZFS, or [gasp] NTFS).
Going back to the 90s--I was using .zip files as a container to keep all files in an order together. As time went on it the requirements included updating some of those files *in a shared access environment*. The workflow didn't permit two people to be updating the *same* file, but simply rebuilding the file on update was not permissible for concurrency reasons (nor was it permissible for speed reasons--I was state-saving each item checked off on a screen.) Solution: The files which could be updated in use were forced to no compression. I then wrote my own code which read the header and got an offset within the .zip that was the body of the file. I then treated it as a simple record-based file with that offset and wrote my changes. The CRCs were recalculated and written if the user left the screen, if the user crashed out (this was the DOS era and the machines were on a factory floor) the file would be reported as corrupt but would actually be intact. My program didn't check the CRC and would perfectly happily handle the situation. Ideal? No. The best that was realistically possible at the time, I believe so.
#5: Why change something that is already working well? The extension is .epub, even if it is a zip file.
You described ePub format, where HTML, CSS, images files go in zip archive.
While we're at it: https://www.princexml.com/samples/
One issue is how precisely EPUB, which is really XHTML, can reproduce layout. What are the possibilities here? The OP's standard is that the document will look "reasonable". The imply that HTML would need new layout capabilities to match PDF, at least for line breaking:
There's two ways to make progress here. One is for browsers to provide more typography tools. Allegedly, text-wrap: pretty is supposed to help, but in my brief testing it doesn't seem to improve line-break quality. The other way is to pre-calculate line breaks, which would only work for fixed-layout renditions.
Also, though the author mentions annotations, I don't see how they intend to implement them.
I don't think this is unreasonable as a solution. By all means let's try to get a smart reader, but letting people create defaults for their documents that can be overridden if desired by the user is a good middle ground.
The EPUB3 standard is much more complex than EPUB2 (media overlays, mixing fixed layout with reflowed, MathML…). In my experience the implementations are much more varying, and most of them aren't complete. So a "Portable EPUB" may not render as expected because the reader tool lacks some specific feature. The author also requires full JS support, which I supose does not help with portability.
It's discussed at the very end of section 8 and all of section 9 that interactive functionality would use web components.
What an ebook format needs is a semantic form of markup, which adapts to devices it is rendered on. HTML + CSS were invented for this goal.
With that, book layout authors should consciously relinquish some control on how the book looks, and hand it to the reader. Slight visual imperfections are a small price to pay for this. Who needs visual perfection should go for a PDF.
This, of course, becomes hard if any interactive stuff is involved. I would suggest that larger interactive elements should open in a dedicated view when needed, and tiny interactive elements should embrace reflow.
HTML (with SVG and MathML) is probably fine for most books, but CSS has spent 30 years resolutely resisting basic typography, i.e. default text baseline alignment.
And aren't annotations (and references) already part of the EPUB specification, and probably even the HTML specification ?!?
Finally, I disagree with the press-and-hold for popup being better than the usual practice of hyperlink anchors, IMHO their jumping around is much less disruptive. (As long as the reader's "return" function is working properly, and/or - for the bijective ones - they provide a "back" hyperlink.)
I'm pretty sure they are not, based on looking carefully a year or two ago, on recent discussions here on HN, and on the OP's belief that they need to invent it.
Interestingly one of the few pages I ever saw it fail on was this article on portable EPUBs. Guess it has too much magic going on to make the formatting work. The saved page is perfectly readable, but the style is nothing at all like what the original page was for some reason.
I like how fbreader on Android just displays all books exactly the same, and as configured in the app rather than using any of the styling from the EPUB file. I never noticed that it tried to apply CSS or run scripts included in files and I hope it never tries to do either of those things. Loading external dependencies sounds like an even worse idea and I did not think that was even allowed.
Edit:
On that note, what's up with the Firefox Add-Ons?
Currently, they are all setup so that to do something interesting, they need all the permissions.
Which leads to a natural market being created, of bad actors go shopping for Add-Ons they can take over.
Can something be done about this? For instance for this "SingleFile" addon. It needs to access the rendered document in the DOM to be able to introspect and save it all to a file.
But why does it need access to everything? Can't it have just permissions:
- "snapshot DOM once"
- "write to a single file"
Now, maybe that’s how it already works, but I have no confidence in it.
I Perma-web anything that I find interesting, after discovering by going back in my notes that half the bookmarks I'd added no longer existed 5 years later.
I don't think I could do half as good a job if it wasn't for your extension.
I owe you a coffee/beer -- actually i just found your donation page, but still a drink IRL if we ever run into each other at a conference/etc.
[1] https://vertis.io/2024/01/26/how-singlefile-transformed-my-o...
I just think market pressure created by the permissions systems is unfortunate, in aggregate. With x thousands of add-ons, bad stuff has happened, and is going to happen. Any improvement to the permissions which could mitigate that at least somewhat, would be nice.
I agree about permissions. In this case it looks like it needs a bit more, since it has some options like enabling auto-save after page-load for tabs for instance. Not a feature I have used, but I am sure it can be useful for semi-manually scraping sites.
> It's a very well thought through article by the developer of Nota, trying to bring EPUB format up to parity with PDF. It's a serious start and they've already written a viewer. In fact, the article itself is displayed in a browser-based wasm port of the viewer (and looks good!).Ha, I think that one of my first HN comments, 10 years or so ago, was how I wanted to be able to save HTML web pages as HTML, not as PDF. I'm sure I didn't explain (or understand) my reasoning well, but it was roundly regarded as a ludicrous thing to want to do. I'm glad to hear that I was just a decade out of sync.
KOreader gives you more control but in a less friendly manner: in addition to choosing specific CSS from several supplied files, you can write your own.
(Also wins for KOreader: excellent OPDS support, and easy self-hosted sync server.)
This feature makes SinglePage unnecessary for any page using this system.
On a side note, what is interesting with SingleFile is that since the file can contain anything including JS, it's possible to create a local "executable" (html) that uses the resources inside the file and does not use a single external file. I have an actual game-like piece that runs locally with bunch of files and comparatively easily transforms into a single self-running "program" that even runs on a mobile file manager with WebView support
> Therefore I decided to build a lighter EPUB reading system, Bene. You're using it right now. This document is an EPUB — you can download it by clicking the button in the top-right corner.
Because, reading this on a desktop browser, I didn't even notice until it was pointed out. It's more obvious on mobile because the header takes up more of the viewport, but it otherwise behaves pretty much like a normal web page.
This is probably a good thing.
For what it's worth, I didn't see (or at least didn't notice) a spinner when loading the doc for the first time like some other people in the comments reported. I did notice it on my phone, but it went by pretty quickly. I'm not sure if that's the WASM program loading and if it only happens the first time you load the page.
This segues into a point of difference I thought the article would mention, but didn't: performance.
A PDF can be optimized so the pages are substantially independent of each other, which makes rendering pages progressive, random-access, and highly parallelizable.
Since as described EPubs are basically HTML its kinda dumb browsers don't open them - but good luck convincing the Chrome/Mozilla bureaucrats
I think another discouraging aspect is HTML CSS are so huge and bloated at this point that few people can implement a "reader" for EPUB/HTML. It's basically "go implement a new browser". It makes one think a easy-to-parse markdown (like Djot) with some extra rendering bells and whistles would be a more likely long term solution
My personal interim compromise solution is embedded everything (CSS, svgs, scripts and base64 images) into an HTML file. It's similar to an EPUB. It's a bit bloated and ugly but with a bit of care it works and naturally browsers (and by extension basically every user) can open it
Unfortunately a user has no way to really know "oh I can download and store this web page offline". It'd be nice to have some thing like a .htmls extension that indicates it's an HTML but it doesn't have any external resources.
Not that long time ago, browsers could not open PDFs as well. Now all browsers come with PDF reader written in ASM/JS. I see nothing that prevents browsers doing the same for EPUBs. There exist browser extensions that do exactly this already. Its a matter of EPUB format gaining popularity.
My mental analogy is, you can also have offline apps on Android. You can specify this in app manifest. But internet access isn't exposed to the user as a permissions.
Like the author says, Google already injects online fonts into the EPubs they generate. Meanwhile PDF is a battle they've already lost
And also probably why we still have to rely on 3rd party hacky browser extensions to be able to save web pages as a single file.
I didn't notice that before, but now I will actively avoid Google generated EPUB files.
Unfortunately, that feature got killed when Edge became a Blink-based browser. But Edge Canary got a new implementation of that feature a few months ago, so it should land in stable eventually.
This feels like the biggest hurdle to me. The author says "Portable HTML generation principle: when possible, systems that generate portable EPUBs should output portable HTML.". I don't think this is going far enough. If the goal is for this format to be everywhere and repeatable then it needs to be standardised and easy to implement a new rendering engine. Relying on webviews doesn't feel like the way forward. The beauty of PDF is that it is incredibly reliable - a PDF from a decade ago still renders the same today as it used to.
I suspect if an effort like this is to get off the ground, the scope of the document needs to be scaled right back. The subset of XHTML allowed should be very limited. The ability to render a document that looks the same everywhere should be prioritised - fixed layout at a fixed page size first, reflowable second. It needs a standard with a comprehensive test suite of documents + render outputs.
IMO actually this is the question the whole effort hinges on.
If the goal is to replace PDF for the uses that require pixel-perfect rendering on every client just as the designer intended, then this approach is dead-on-arrival.
But if that's not the goal, then that has to be extremely well-communicated by the project, so that people who need that know they need to stick with PDF. Indeed, the project needs to explicitly say that it's not a goal, and that clients should be free to make reasonable rendering decisions within certain specified bounds.
But then we 'needed' magazine-like design/layout. Still, it was document based, so actually pretty good.
After that, we tried to shoehorn HTML to an application distribution platform. Current css layouts are (finally) more like traditional layout engines for applications.
The last 20 years i.m.o. was pretty much a waste of effort because there was no proper way to distribute (cross-platform) applications, well besides java...
It still is.
> But then we 'needed' magazine-like design/layout. Still, it was document based, so actually pretty good.
Styling is not handled by HTML. It's a separate concern assigned to CSS. For convenience HTML offered default styling.
> After that, we tried to shoehorn HTML to an application distribution platform.
It's not shoehorned. It's the use case: render documents. A document is a tree of ui elements. It's the same with GUI frameworks like Qt or WPF.
It still is but you are missing the point of the thread. HTML still is hypertext, some links, some images, some tables. No doubt about that. But HTML is also so much more than that. The spec is a beast. Anyone who wants to implement an HTML based reader has a mammoth task in front of them. It's like "go implement a new browser" like someone said in this thread above.
> "Styling is not handled by HTML. It's a separate concern assigned to CSS."
Missing the point again. We know styling is not handled by HTML. The point of the thread was to tell how big of a task it is to create your own HTML based reader. If you want to create your reader like it or not you have to implement support for CSS too and that too is a mammoth spec.
So our only options are: A. Go implement a new browser. B. Use something like Webkit. C. Implement a small subset of the HTML and CSS specs.
That's true for basically any non-trivial document rendering format. For example, take a look at the PDF spec. Even basic things like parsing the document format is a formidable task. HTML in comparison is a trivial format. The same goes for technologies like TeX or even Microsoft's own Word format, which Microsoft famously had lots of problems supporting. It is a hard problem for all formats, not just HTML.
> The point of the thread was to tell how big of a task it is to create your own HTML based reader.
You're confusing some things. A document format is one thing, but a renderer with specific capabilities is an entirely different thing. You're commenting on the document format and the styling and layout system, and now you're shifting the conversation to what it takes to implement a renderer.
Debates on document formats are entirely separate and orthogonal to debates on how to implement renderers. Renderers for the most trivial things are tremendously complex. There are a myriad of good reasons why we're seeing GUI frameworks built on top of webviews in spite of all the complains about the formats that webviews support, and in spite of the myriad low-level rendering frameworks already available.
To understand the poiny, try to think through the requirements list to implement a renderer for Markdown. It's a document format with a half dozen of features. Would you call it trivial?
If you follow the comment you replied to the discussion was about implementing a renderer. So no, I am not shifting the conversation to implement a renderer. The conversation is about implementing a renderer. That it is incredibly difficult to do today with the modern specs is the point.
It in-fact was, and to some degree still is. I assure you we achieved a hell of a lot of styling before css existed, and for some time after it did but before most of us were using it (much), using features of HTML, some of which were explicitly there to support styling.
I don't think that's true. In the very least, you can use a WebView and feed it regular HTML. If the whole industry uses webviews for GUIs, it's hardly a stretch to use one to render Epub docs.
I guess that's why the article is actually an epub opened with a WASM epub viewer :)
While browsers don't provide a convenient UI for opening EPUBs, they should have no problem rendering the chapter HTML files contained within.
In the absence of browser support, writing a server-side EPUB-to-browsable site proxy that adds chapter navigation controls and simple layout options shouldn't be too difficult.
Incorporating the necessary DRM support required to view the majority of commercial ebooks through such a proxy would very likely be legally problematic, of course.
Come to think of it, any form of publisher-approved DRM EPUB browser support sounds like it'd be about half a technical step away from DRM support for web pages in general, which is a horrifying prospect.
If you read the article, then you just did open an EPUB.
I was surprised when the author mentioned iBooks doesn't support scrolling view, so I tried it myself. Turns out iBooks on macOS does not support scrolling for ePub files, but it does on iOS and iPadOS. Very strange decision by Apple.
1. https://googleprojectzero.blogspot.com/2021/12/a-deep-dive-i...
I have never seen any kind of technical documentation published in any other format than PDF that is comfortable for reading and searching, even when that is done on a mobile phone.
I do not want a document that changes appearance depending on the device used for reading or depending on its temporary state, like window size. I want a document whose layout has been well conceived by its author and which is fixed, regardless of what I happen to use for reading it.
When I happen to read it on a smaller screen or window, except for trivial text-only documents, I do not want changes in layout, but I only want a smart reader, with comfortable means for fast zoom and pan, and which does not have stupid behaviors (like some Android readers), for instance where scrolling vertically (including Page Up/Page Down) also moves the document horizontally (preventing the easy reading of a column of text).
The traditional recommendations for the maximum width of a text column are good enough, if observed, to ensure comfortable reading even on a mobile phone. Only when the author breaks the traditional typographic rules by making extra-wide columns, the reading on a mobile phone becomes inconvenient.
On my phone I have to either zoom in or turn on landscape mode (which usually means turn it on globally, I can't do it just for the reader app).
On kindle, a full page has too small font due to so much margin, and fitting the width shows me 80% of the page, and then I have to scroll down for the last 20% and my eyes have to find where exactly I was reading.
However it does sound handy, I kinda want a dedicated tablet for sheet music.
Would I prefer more content to be reflowable etc.? Yes - but with a tablet it isn't strictly necessary, just nice to have.
> view in landscape or pan and zoom.
This is awkward, that's the issue I mentioned above, how annoying is to have to do that if you're reading for a 30-60 minute session.
forced-auto (overrides most lawful-evil apps that try to force an orientation for not good reason)
auto-portrait (allows only the two portrait orientations, auto-landscape (similar, good for some games & video apps)
lock-current (for chaotic-evil apps that lose state or rebuild the whole UI in response to rotation events).
Can you provide an example of what you mean? My experience is completely the polar opposite.
These are concrete examples of documents that I might have read during some flights or when waiting for some flight, on a smartphone.
When reading something like a fiction novel, reflowing the text based on the window width may be acceptable.
On the other hand, the navigation through a huge document half of which are tables, figures, diagrams, schematics and graphics is extremely painful when it is in HTML format so the layout changes based on the device and window used and there are no means to jump quickly e.g. to page 1436, then to page 2117. When zoom, pan and scroll are correctly implemented, which unfortunately happens seldom, they are much less distracting than the random changes in page layout caused by rendering as done by a browser.
I strongly dislike whenever a company provides only a Web documentation that is hard to navigate, instead of also providing a PDF manual.
Web documentation may be acceptable for very small documents, but not for most of the current technical documentation, where many thousands of pages for a manual are common.
Perhaps an EPUB format extended with everything necessary to completely describe a fixed page layout might become competitive with PDF, but I will have to see an example to believe it.
For now, whenever I see a book or any other document both in PDF and in EPUB formats, I always choose the PDF variant, because without exception it provides a better quality of the rendered pages.
I've been boosting the idea in the OP, but more for things like "your local council's meeting minutes" or "your English class assignment" or "a research paper".
Though I do want to point out that even moderately complex specs, when designed for the web, can work well. For example, the HTML spec doesn't reference page numbers, but has extensive internal hyperlinking: https://html.spec.whatwg.org/
> Perhaps an EPUB format extended with everything necessary to completely describe a fixed page layout might become competitive with PDF
I highly doubt this will ever happen, for use cases which require fixed layout. But there are plenty of use cases where fixed layout is unnecessary and inferior.
Even many of the PDFs created by scanning printed documents allow reasonably reliable search/copy/paste, if they had been processed by an OCR.
Are you sure about that? As far as I understand, extracting text from an ultimately vector-graphics-like PDF heavily depends on ORC-like heuristics on the PDF consumer's side.
The ToUnicode mapping table can help with the glyph-to-codepoint mapping aspect of this, but figuring out the difference between the gap between two letters and two words seems hard.
I've seen bothtypesofissues mentioned in the following article i n t h e p a s t, including in a specification document I use multiple times per day for my job:
https://web.archive.org/web/20220328102205/https://filingdb....
Nevertheless, I have been using very frequently every day for many years search and copy + paste from PDF documents without any problem. I usually prefer to use mupdf as the PDF reader, because it is very fast (it also works better as an EPUB reader than the other EPUB readers that I have tried), but there are some seldom-encountered PDF files that mupdf cannot parse, in which case I fall back to other PDF readers, e.g. okular.
The only case that I encounter when search/copy/paste does not work is in scanned books that have not been OCR'ed, so they contain only bitmap images of the pages, without text.
The problems mentioned at your link are caused mostly by the PDF specification being too permissive, which allows abuses like using a non-standard character encoding coupled with the use of a non-standard font. However, this specific type of abuse could not be prevented by any specification without using some sort of AI to decide whether the glyph used for a character encoded as Unicode "A" is really a kind of "A".
Among the problems enumerated at your link, I have encountered a few times the case when there are thin spaces inserted between each letter of a string. In such a case it is annoying to remove those spaces after pasting the text in another document, but this is something that I have seen only very rarely.
I don't jump to page 2112, I use the table of contents to jump to section 3.1.2, which is as fast if not faster.
I totally see what you mean for circuits though!
There are: OpenXPS (https://en.wikipedia.org/wiki/Open_XML_Paper_Specification) or DjVu (https://en.wikipedia.org/wiki/DjVu).
So it seems to be a bad idea to try to have a one-size fits all standard : we're much better off with two digital document standards : one with full multimedia and interactive capabilities (short of networking), and another, a subset of the previous with the limitations like : monochrome, no multimedia, interactivity mostly limited to (still in-document) hyperlinks...
And guess what, we already have two formats that are almost there ! HTML (see also MHTML=EML) and EPUB.
(And of course a 3rd one for physical archival and the rare digital fixed layout documents, for which PDF/A already seems to be decent enough.)
If that really is the only thing that differentiates an EPUB reader from (a subset of) a browser, it seems that the simplest and least-effort thing to do would be:
1. Enable existing browsers to treat a zip file like a directory, under certain circumstances.
2. Describe a machine-checkable restriction of HTML that guarantees fully local assets as described in TFA (this might be as straightforward as requiring all URLs (except for links) to be relative). Declare any zip file that satisfies this restriction to be "a portable EPUB".
3. Forget about special-purpose EPUB readers altogether, both existing and new.
As for fully local assets--what I would like to see is a requirement that all objects have locally available fallback options. You can't get that font, use this standard one. Might not look perfect but it should work reasonably well if you limit yourself to things in the fallback.
There is still one failure mode he didn't fully address--images that don't work right. Fundamentally, this is because images can't reflow. To a large degree I think this could be addressed with using SVG, but that's not going to handle things which are truly raster rather than vector. Unfortunately, I think the only way to address that is provide a few resolutions.
(And I have faced a related problem that I haven't seen a good answer for but I haven't looked hard: SVG that is aware of it's resolution and can bring in more detailed images if it has the resolution to do so. Try to draw sub-pixel lines and you're going to get a blur, but blowing up the same image and those lines become a good thing. Windows acknowledges this with icons, you can include a variety of resolutions and it will work with whatever is most appropriate for the context.)
If we instead mandate that every font used is either packaged within the zip, or one of a small, fixed, well known list of "standard" fonts that every system can be required to support, what would be lost? (IOW, what do we gain by allowing the concept of "fallbacks"? Because what we'd lose would be consistency between internetful and internetless renderings, and self-containedness.)
>images that don't work right
What's an example where storing a single high-resolution image, which can be dynamically scaled down if needed, doesn't work well?
As for scaling--it doesn't work when the zoom is too great. It really doesn't work too well if you have some sort of texture to your image and you squash it too much.
Consider a real world case: maps. I'm looking at a topographic map of an area around here, the farther back I pull the camera the more features get omitted because painting them would end up obscuring more useful information. You could use lines with a wider spacing but that isn't even too useful as the camera pulls back.
Works of art. True love! That's very high praise.
This whole ethos on readers learning is in stark contrast to books that feel like the authors show off how smart and clever and profound they are instead of caring about their readers comprehension. I include very highly praised books even on HN.
[1] https://www.plai.org/#direct-links-to-the-tutor
N.B. The author of this post, portable EPUBs, also works on language learning. A whole ethos... https://arxiv.org/abs/2401.01257
A PDF can include the fonts. But it often doesn't, and relies on system fonts. One reason for that is because including fonts in the PDF can dramatically increase the size of the file. In some cases a single font could be larger than the entire rest of the file. I've also worked on implementing embedding fonts in some software that generated PDFs. It was surprisingly difficult to figure out how to get it to work reliably.
> PDFs are rendered consistently.
Not as much as you would think. There are several cases where the same PDF will render differently depending on which PDF viewer you use. Usually the differences are pretty subtle, but occasionally there are edge cases that result in pretty significant differences. I've even run into a case where the same version of Acrobat reader will render a PDF differently depending on what OS you are using.
Normally all PDF documents include only the glyphs corresponding to the code points actually used in the text rendered with that font.
That is why you can go for instance to any site of a vendor of fonts and you can download freely a PDF sample text of an expensive font. You can easily extract the font from the sample PDF, but it will be useless, as it will contain only the few letters that had been used in the sample text.
found this out, after 20-something years of consistent pdf renderings, in a job interview because my docs allegedly looked odd :/
the daily wtf ...
Except for the small number of standard system fonts, for the other fonts the PDF document normally includes only a small subset of their glyphs, corresponding to the characters that are actually used in the text that is to be rendered with that font.
Example note: "EPUB originally didn't support remote resources and people put a lot of work into changing. Loading stuff over the network is HTML's killer feature. Blocking network assets is a setback for format adoption, not progress."
> Blocking network assets is a setback for format adoption
People are standing here telling him that _allowing_ network assets _is a setback for format adoption_ and he's just going to keep pounding this stupid, obnoxious drum of his until he runs everyone off.
> almost all of the problems described would be solved by getting OS vendors (Google, MS, Apple) to invest more money in EPUB
That's back-asswards. Google, MS, and Apple don't give a shit about EPUB, they never will, and it's arguable that we're better served with them not buying a seat at that table considering how poorly their "help" has helped web standards, as much of the rest of his dismissive thread helpfully notes.
If he wants money for EPUB standards he should shake the cup at IDPF members who rely on it, and particularly Amazon, to whom he quite vocally abdicated the publishing space 12 years ago.[1]
Barking at operating system companies is nonsense at best and how we wind up with another, even more avoidable situation where the space is held hostage by them at worst. At least Amazon can chuck some goodwill money at EPUB development while continuing to kick its ass up and down the market with MOBI.
(Aside from all this, his dismissals of the "clunky" reading system complaint, citing how EPUB has "too many divergences", only further proves to me how tunneled the vision is of the people involved. To hell with forking or improving EPUB, then, because it can't be improved if that's the attitude of the people most involved with or influential within it. What bloody point is there in the customizability of a format that _nobody_ can effectively build tools for or consume?)
1: https://www.baldurbjarnason.com/notes/amazon-wins/, in which he also admits that he has no idea how to work with IDPF, which is a really great sign of how long things have been going this badly in this space
https://www.w3.org/groups/wg/epub/former-participants/ certainly shows Google and Apple participated in the working group. Apple also has a book store selling EPUB books and has EPUB readers for both MacOS and iOS. Google also has an app that handles EPUB.
> People are standing here telling him that _allowing_ network assets _is a setback for format adoption_ and he's just going to keep pounding this stupid, obnoxious drum of his until he runs everyone off.
I don't know the background that you're frustrated about, but I'd suggest that the answer might be: 'it depends' - and it depends on the intended purpose of the format in question. PDF is self-contained, and can be read (mostly) reliably on almost any device with the right software; PDFs having to have internet access to be read or opened would be a bad thing; further, the same goes for most formats - including EPUB (as you say) and audio files, picture files, etc.
Article: "A PDF is a single file that contains all the images, fonts, and other data needed to render it."
This is only true if you are using PDF/A, or have explicitly bundled all of your fonts in some version of the PDF standard.
Otherwise, 14 total typefaces must be rendered by the viewer. These 14 are: Times (in regular, italic, bold, and bold italic), Courier (in regular, oblique, bold and bold oblique), Helvetica (in regular, oblique, bold and bold oblique), Symbol, and Zapf Dingbats.
The 14 standard typefaces can vary between viewers:
https://en.wikipedia.org/wiki/PDF#Text
"...the base fourteen fonts... or suitable substitute fonts with the same metrics, should be available in most PDF readers, but they are not guaranteed to be available in the reader, and may only display correctly if the system has them installed."
Depending upon what the viewer bundles, PDFs using these 14 might not render as expected.
Below is a deeper discussion from the wiki:
https://web.archive.org/web/20110718231502/http://www.planet...
I'd value that a lot more than Pocket (which I always turn off).
Lots of governments around the world would love to know what their citizens are reading. Few would be bold enough to go after this directly, but if some company operating in their country has the data then there’s a path for that government to get the data.
Don't confuse what's necessary for standard fiction books with what the format should support.
JS and interactivity are fine, in technical books, reports, or niche fiction.
What I absolutely agree on is that epubs don't need is networking. Resources on the internet get stale after years or decades anyway, so inclusion of any network assets into an epub guarantees that the work will degrade over the years. References can be web links, but nothing from the internet should be embedded.
a) What epub editor software allow direct editing of MathML, and is it a good idea to depend on such software being available in order to edit the book without tinkering directly with the MathML by hand?
b) Does MathML support everything mathjax does?
And it is a feature of books that they stay the same.
Both can be a feature, the ability to change (e.g. they fix something in the cloud), and the disability to change (e.g. you can have it as you bought it).
See: https://w3c.github.io/dpub-pwp/publishing-snapshots/FPWD/Ove...
It's true that running code in the document has some downsides, but the vast majority of people does it all the time in their browsers. And it comes with tremendous upsides. Just imagine large amount of data presented in interactive tables which can sort, filter and export or interactive graphs inside the document. We already use HTML+JS so much, why should we stop at documents? Yes, they can't be printed, but in my observation less and less people even own a printer these days, and I see no reason why this trend should not continue. I bet the future will be mostly living, interactive documents.
It's funny that I just mentioned this in the other thread [1], but I also felt that there is a need for a format that is self-contained and widely supported by standard software (by which I mean browsers). A well-specified open format would be great, but until then I tackled the self-containedness problem with JS and wrote a Python script that zips and bundles all assets and embeds them as a SPA into one HTML file [2]. The focus is on Sphinx docs but it should work in general with all distributed HTML docs.
This probably has to do something with them having nothing to do, as the big companies managed to convince the frontend dev community that the single best thing to generate layout is on the client machine on the fly. Of course they did it so the users will have a hard time selectively blocking the layout scripts from the ad/spyware most contemporary (web)software development is about.
This led us to the point where saving (or God forbid printing!) an article needs a lot of effort in many cases.
My observation is: when I need to go to work on the field, I need printed documents. Printed documents don't need firmware updates, their batteries don't run out, and no, I don't need interactivity in documents.
Self contained HTML is a good - and necessary - step, but interactivity and executable code is not something we usually need in documents, I only saw somewhat legit need for it on corporate abomination of documents (and some teaching materials possibly).
If quick high-resolution referencing (page x, yth paragraph, zth line) is necessary, I think the way to handle it is to reference a phrase on that line ("the cat ran"...), which the reader can search for. If the search interface is lacking, that's an epub reader failure, not a format failure. Or, if the search option is considered insufficient (it does require typing with a [virtual] keyboard), paragraphs can be numbered—as many ancient works or works in translation already are, because such works have many editions and can't be layout-exact copies of each other.
If paragraph referencing is necessary, visibly styling the paragraphs with numbers helps dramatically. There's no reason it has to be exclusive to high-profile ancient or classical works, like Plato [1].
Classic poetry and plays, where referencing needs to be most exact and fast, already tend to give up any hope of everyone using the same edition, and simply avoid flowed text and then number the lines.
I believe that paragraph referencing is, by far and without any contest, used primarily by any text subject to reviews and revisions. This means technical reports and academic papers.
All academic papers I subjected to review were forced to use templates that enforced paragraph numbering. Even though each version of those documents were only read by a dozen readers or so, all papers submitted to those journals had to use the template. This means hundreds of documents (see half a dozen revisions per paper submitted per each edition) had that hard requirement, and this took place for each edition of a single journal.
For instance, if I highlight in Chapter 8 "In 539" [next paragraph] "Belisarius" [next paragraph] "marched on Ravenna" [10 paragraphs later] "In 540 Belisarius entered Ravenna".
I would like to export this with the Chapter header and detailed highlight locations OR just as one sentence with subtle links to the locations.
There's some interesting news in this space: next version of Zotero will have epub annotation (WADM) support: https://forums.zotero.org/discussion/110487/embedding-annota....
The third way is to develop non-browser clients, like the author's own Bene. While it currently uses Tauri and therefore the system webview, there's no reason it should always do that, or that another client couldn't be developed with a focus on typography.
Want your documents to look super nice with all the kerning and line breaking your heart desires? Get a proper reader app.
Just want the content? Sure, open it in a browser or a basic reader.
At the same time the top comment is suggesting EPUB should not require Javascript.
I never use the HTML markup, CSS, etc. when I read an "ebook". I dump it to text and read it with less(1). I do not need to see images. I just want to search and read text. I use stupid simple shell scripts; something like the following.
#!/bin/sh
printf "(\n"
for x in ${1-*.epub};do
echo printf \'\\n\\n,,,,,,,,,,,,,,,,,,,,, "$x" ,,,,,,,,,,,,,,,,,,,,,,,\\n\\n\'
7z l $x|sed -n "s/.* /7z x -so $x /;/nav.xhtml/d;/[tx]ml$/p"|sort -n
done
echo ")|tr -cd '[\\\12\40-\176]'"
epub.sh 1.epub |sh > 1.htm
links -dump 1.htm > 1.txt
less 1.txt
epub.sh 1.epub > 1.sh
vi 1.sh
1.sh > 1.htm
links -dump 1.htm > 1.txt
zstd -19 1.txt
zstdless 1.txt.zst
To be fair, Epub has been largely ignored and neglected by everyone in the world. Virtually no reader supports rendering math notation, and virtually no significant publisher on earth, including the likes of Arxiv, offers Epub downloads. Commercial publishers force DRM onto every format, which excludes Epub, and non-commercial either stick with PDF or don't care.
Though I'm curious whether the clunky old-but-still-living HTML (especially in its ugly XML variety) + CSS are the right foundations for the portable format of the future? Since the author has also developed the whole new document language would be nice to read a more in-depth overview on that subject. Or why limit to the ugly duckling of JS in the future when WASM exists?
> content by pages. Instead, we should mandate a consistent numbering scheme for block elements within a document, and have people cite using that scheme.
that's indeed the proper and more precise approach, though we could still have those "fixed layout epub" pages as a backup coordinate system
That said, epubs are great for reading books on mobile. The advantage for pdfs is that they contain highlights/notes, so you can directly import them into Zotero and all your annotations are there. For epub, you have to hope there is a way to export the annotations that are stored by the reader app, and then you have to process them further. Readera is a great reader for mobile that makes this possible. I'm currently working on a script that will convert an epub to pdf, extract the annotations from Readera, and mark them in the pdf. Then I can import the pdf into Zotero, while still retaining the great reading experience of epubs.
One of the nice benefits I can already experience in his document it the working TOC sidebar which allow navigation in the document. (Compared to classical HTML not PDF)
Images, limited to things that need grid-of-pixels representation like photographs, should be limited to that.
How I feel about this depends a lot on how much I trust the people who created the document and/or the person who sent it to me. I would trust a website I specifically selected with a defined TLS web of trust more than I would trust a random spam email.
When we think about the risks associated with the complex and inherently insecure format known as HTML we tend to assume the level of trust available on the web. If we package up a bunch of HTML in a standalone document then we lose that assumption.
One stunning failure of current ePub technology is the lack of publication-quality justification and microtypography.
Currently, PDF authors are overloading "Tagged PDF" annotations, which were initially designed for accessibility features such as high contrast, large text, or screen reader (text-to-speech) document flow. It's not a general solution to reflowable PDF output, but it's better than nothing, perhaps.
Adobe realizes that such additional markup does nothing for the vast universe of existing PDFs. They have trained an AI model to infer document structure and create the tags on the fly; the marketing term for this is called "Liquid Mode" in recent versions of Adobe Acrobat Reader.
I'd like to see a reflow engine in TeX, and there are some good implementations
https://hint.userweb.mwn.de/hint/format.html
The CSS Text Level 4 has a `text-align: auto` attribute. They point out that text layout is complicated, various human languages and scripts have different approaches to adjusting text width, some of which can be generalized.
https://www.w3.org/International/articles/typography/justifi...
ePub renders wildly different on my eReader (kobo), my linux laptop (various apps), my iPad and my iphone. And i'm not talking about screen sizes, i'm talking about various elements being rendered largely incorrectly (and there's a matrix of incorrectness across implementations.
PDF documents on the other hands... they render just right, everywhere. I have to zoom and scroll, but I will never have to ask myself "will I be able to actually read this document?" when dealing with PDF.
Oh and by the way... I still print stuff from time to time. Yeah way less than it was needed in the past, but it's still a necessity. Can you even print stuff (sections? pages? selections?) from an ePub?
> Bene is designed to make opening and reading an EPUB feel fast and non-committal. The app is much quicker to open on my Macbook (<1sec) than other desktop apps.
This is elitist at best. Claiming something "is fast" on top-class hardware is misleading at best (if not dishonest).
Try and running that on low class hardware (stuff like chromebooks but also laptops from at least 7-8 years ago) and let's see if it's still "fast".
I'm not convinced.
To me it feels the other way around, if I have to zoom and scroll, they render just almost right.
And that almost is extremely important for actual reading. If you're quickly skimming a PDF, sure. But to sit down and read for 30 minutes? One hour? Fuck zooming and scrolling. My kindle just displays a page, and I tap it and it goes to the next page. Can't really get a much better reading experience than that.
Epub on ereaders work well but only if you’re reading fiction. Most images and almost all tables and charts have been messed up by epub rendering anyway. And ereaders are black and white, so you’re losing information anyways.
Tables and charts are also a specific use case. There's no mention of them whatsoever on the website so one can assume this is talking about mainly text.
In other words, you described the only time when PDF are more comfortable: if you have an iPad and you need to read charts/images/tables. Far from the claim of "they render just right everywhere".
Most e-readers are "low class hardware". ePubs work fine on most of them.
In terms of performance there isn't a clear winner here; both can be somewhat slow at times for large documents, but are also "fast enough" for the common case, even on older low-spec hardware.
I do think the general software ecosystem surrounding ePubs is not quite there yet, but that's mostly a matter of UX and "software that hadn't been written yet". As a format ePub is the clear winner for many (not all) scenarios. I struggle reading many PDFs because "zoom and scroll" that you mention is a right pain if you have to constantly do it (which you often do if you zoom text). Comfortably reader PDFs on my phone or e-reader is basically impossible.
https://www.russellbeattie.com/notes/posts/the-decades-long-...
(Which still hasn't been posted as a "news" it seems... should I just submit it myself ??)
I am glad about this, I do not want to download a document and it require any input. a document should be a document, nothing more. If I'm getting a book to read (pdf), I expect a book, not a webapp.
Also, most dedicated reading systems (Kindle, Kobo, etc) don't allow javascript, which means your components will not work. That might of course change, but I wouldn't hold my breath for it.
"less semantically rich" than what? Web pages? Or less rich than PDFs, which is what he's actually proposing to replace?
Yes, that’s what the author is proposing we do.
On the other hand: this looks awesome for the Web. Mix it with a blog platform like Medium, and it will improve my Web browsing experience tenfold.
If I use, for instance, CloudConvert [1], I generally get a document that gets flowing text roughly right, but still interrupts the text with page numbers and book titles (that were originally at the top of each page) and includes additional bizarre line breaks, etc.
Every so often I wonder if this is an LLM problem ("please reformat the following text to...") but I think that one shouldn't reach for an LLM for these kinds of things.
I wonder if we could instead look at religious texts such as the Bible (e.g. John 3:16) and code editors (e.g. Ln 4, Col 12) for referencing locations in reflowable text. The same way you can jump to a footnote in a document should allow you to have an actionable reference to a specific location anywhere in the text. But I don't think the text should be stylized like how the Bible has the numbers (e.g. 16) scattered within the text itself. Those should probably be hidden within the text and leave the reading software to display the first line number of the page down at the bottom instead of the page number. That might look like "4" the same as it currently does, but this 4 references a section of the text rather than a page number. Perhaps it could be togglable for greater detail and display word 23 of section 4 as "4:23". Or maybe it could consider the chapter too. For example, chapter 2 section 4, word 23 would look like "2, 4:23". This might get funky in a Terry Pratchett novel, but it would hopefully allow for easier discussion of exact parts in a document and significantly easier linking.
I love the interactive code example for marking up The Rust Programming Language. That gets at what I was saying above although is more targeted at a document than referencing parts of a novel.
Kudos to author for creating Bene[0] as part of this proposal. That was cool to discover I was using their tool to read the proposal itself!
It is quite complex already, and making it into portable document format will just make an order of magnitude more complex.
Another single file book format that used to be popular many years ago was chm, that had its own security problems. Maybe adding the possibility of executing (js) code is not the best for something that should be mostly static, and css be used to enable some level of safe interactivity.
Using HTML as a base has a lot of sense.
#!/bin/zsh
#
# Extract epub file to a temp directory, launch shell to edit it, and re-zip
# it. Nothing about this is really epub-specific as such.
echo " $@" | grep -q -- ' -h' && { sed '1,2d; /^[^#]/q; s/^# \?//;' "$0" | sed '$d'; exit 0; } # Show docs
[ "${ZSH_VERSION:-}" = "" ] && echo >&2 "Only works with zsh" && exit 1
setopt err_exit no_unset no_clobber pipefail
full=$1:a
tmp=$(mktemp -d)
bsdtar xf $1 -C $tmp
cd $tmp
print "Editing $1; press ^D to exit"
zsh ||:
mv -f $full $full.orig
zip -f $full *
cd -
rm -r $tmp
And that is why PDF sux for reading on the phone. And why epub is massively better if you want to read articles and books.
Bene seems to be in alpha stage.
Publisher and editor laziness may be a reason to be cautious about epubs currently for niche or esoteric works, but that's not the same thing.
> I bought a book set in the late Middle Ages which managed to transcribe all “þ” as “p”. Until publishers care...
The book market these days makes it challenging to do high-quality editing up front for republishing niche books in a new format. Publishers try to cut corners, outsourcing epub conversions to people who don't care and don't know what they're doing, or they OCR it, have an in-house editor (who also doesn't have a personal affinity to the subject) give it a once-over (maybe), and release it.
Now, there are some attempts to fix this situation by Xe(La)TeX and Lua(La)TeX, but since TeX seems to be so much tied to PDF these days, it should probably just be abandoned by most scientific publishing in favor of the likes of GNU TeXmacs (note : it's NOT TeX in GNU Emacs) and HTML with MathML.