smoores

Yup, great point. Especially if you've used some tool to convert from another file, like a PDF, into an EPUB, you can easily end up with the entire book in a single XHTML file, which, again, can be pretty heavy for a browser to parse and format! I also have no idea whether Calibre et al actually use native web views, or have their own renderers, which are almost certainly less performant than native web views!

It was, actually, very enjoyable! When we pulled React ProseMirror[1] out of the NYT text editor, it was a pretty laborious process that we had to careful plan and execute for months, and we still ended up with an internal fork for a while.

By contrast, this was mostly just moving a file around and then writing documentation and cleaning up the public API. I rather enjoy thinking about and modeling library APIs in general, so I actually had a lot of fun with it!

[1]: https://www.npmjs.com/package/@nytimes/react-prosemirror

Thanks! Absolutely. You can invite users to your Storyteller server and give them whatever permissions are appropriate (e.g. you can choose whether they can only download books, or can also manage uploading and syncing books and/or managing users). It has SMTP support for emailing invites, or it can just generate invite links for you to share yourself.

More info here: https://smoores.gitlab.io/storyteller/docs/administering#inv...

Of course! I'm hoping to have a web reader with Media Overlay support built in to Storyteller available in the next few months, along with some much needed library management tooling, so maybe that will be useful for you! I'll try to make it snappy :)

I would guess there are multiple potential pitfalls here. Firstly, not all ebook formats are created equal -- Storyteller only operates on EPUB files, because EPUB is an open source format and it supports Media Overlays (read-aloud) natively. I can only really speak to that format, but there are others (MOBI, PDF, etc).

An EPUB is just a ZIP archive of XML and XHTML files (plus other assets, like images). Partly, I suspect, because of the dearth of actively maintained open source projects in the space, and partly because of the nature of tech in the book publishing industry, EPUB generation software used by authors and publishers often messes up this spec, which means that EPUB readers sometimes need to have fairly complex fallback logic for trying to figure out how to render a book. Also, because EPUBs are ZIP archives, some readers may either unzip the entire book into memory or "explode" it into an unzipped directory on disk, both of which may result in some slowness, especially if the book has lots of large resources. The newest Brandon Sanderson novel, for example, is ~300MB _zipped_.

Additionally, and perhaps more importantly, EPUBs (and I believe MOBIs as well) represent content as XHTML and CSS, which means that readers very often need to use a browser or webview to actually render the book. Precisely how they deliver this content into the webview can have a huge impact on performance; most browser don't love to be told to format entire novels worth of content into text columns, for example.

Outstanding. Let me know if you run into any issues!

Hahaha, well if it comes to you, the library will still be there for you :)

Right, this was actually just a file within the Storyteller web package to start. It was fairly well defined, and so pretty easy to pull out into another package in the monorepo, but Storyteller is the primary consumer at the moment, and I want to be able to develop them in sync. Plus, it provides a great test bed for development of the library!

edit: I forgot to mention that the eventual goal is to (hopefully!) publish this package as @storyteller/epub, along with any other packages that end up split out of Storyteller. That will probably include at least a @storyteller/synchronize and a @storyteller/cli.

Unfortunately, someone seems to have snagged the @storyteller org on NPM several years ago and left it to languish without really using it, so I'm waiting to see whether GitHub will consider this squatting and transfer the org to me.

I've also tried reaching out to the developer that owns the org, but they don't seem to have been active on GitHub or NPM for the past 5 years or so, and my only real strategy for reaching out to them was to open an issue on one of their other GitHub projects!

Oh, hello, I didn't realize this was shared here! I guess let me know if anyone has any questions. I mostly wrote this piece as part of processing some hard feelings I've been having and seeing shared among Jewish folks around me, but I also ended up learning quite a bit about phonetic encoding algorithms, and I've spent several years at this point steeped in forced alignment via Storyteller.

It's true, "sure" and "shore" are not pronounced exactly the same, and accents absolutely can vary, which is part of why Beider-Morse produces multiple encodings for each word. But the goal of Soundex-style phonetic encoding systems isn't to perfectly encode a word with a precise alphabet like the IPA. Rather, they intentionally introduce fuzziness so that words (really, names) that are pronounced similarly will be encoded the same way.

Perhaps "sure" and "shore" was a bad example; it's tricky to come up with these! And you're right that the encodings that happen to overlap for those words are technically "incorrect" pronunciations; again, these Soundex-style encoders are designed for surnames, not general English words. Some Storyteller users are testing out a version of Storyteller using this encoder to see if it makes any improvements (so far it seems like it's not worse, but not necessarily better!), but I won't be surprised if it doesn't end up making it into Storyteller long term.

Mostly I wrote this piece not to advocate for using BMPM to support forced alignment, but as a way to express the emotional journey that I found myself on as I learned more about these systems and where they came from.

No problem, glad I could clarify!

This press release elaborates on the motivations for the strike: https://www.nyguild.org/post/new-york-times-tech-guild-votes...

Oh, this is because the page linked above is for the “Times Guild”, not the Time Tech Guild. Times Guild is the journalists; the Times Tech Guild is the product engineering folks that just authorized a strike. They are two separate unions within the Times. They support each other but do mot have precisely the same platforms or negotiations with the company.

This press release goes into more detail as to the motivations for the strike: https://www.nyguild.org/post/new-york-times-tech-guild-votes...

I’m not sure exactly what you mean, but most non-managerial employees at the Times are now unionized. It is no one’s goal to improve the labor standards of the union members at the expense of non-union employees. It is nearly impossible to obtain contract language covering anything other than workplace and employment conditions for union members, as the company isn’t legally required to negotiate over any other topics, which is why you’re seeing reference to requests “for members”. That said, it is very often the case that wins for unionized employees benefit non-union workers as well, because it’s almost always not worth the administrative overhead for the company to have multiple separate policies for similar jobs.

Kathy isn’t a people manager, the Times just has confusing titles for its data folks. Product managers and project managers likewise often have “manager” in their titles without being people managers. No one with direct reports is allowed to be a member of the union!

Yup, this is the goal! Library management and a reader app already exist, though there's definitely work to be done, especially on the library management front.

There's an open ticket for languages other than English! https://gitlab.com/smoores/storyteller/-/issues/10. If you want to take a look, please do! I don't have any contributing guidelines yet, unfortunately, but they'll probably come soon. I think Whisper does have Cantonese and Mandarin support, so it should be possible to add support for those languages, though we'll have to look into nltk support for sentence tokenization as well!

Thank you, that's so exciting! Please let me know if you try it out, I'd really love to get your feedback!

Wow, this really blew up while I wasn't looking! Thank you everyone who's popped in here to ask questions and give feedback. If anyone does spend some time trying to set this up, please don't hesitate to hop into our Gitter channel (https://smoores.gitlab.io/storyteller/docs/say-hi) and say hi or ask for support or give feedback.

That's a great point; I'll try to add some more of these. I definitely meant to link to the app store page from the docs; I actually just updated them to include that.

It's a full ebook/audiobook library, with sync, though I've focused much more on the reader experience so far than the library management experience. Improving the library management experience is on the horizon, though!

Not yet, but it's on the roadmap (https://gitlab.com/smoores/storyteller/-/issues/13)! This one actually ought to be pretty straightforward; the only trick to it is making sure that it doesn't interfere with the local-first goals of the apps.

Thanks! What's your preferred installation/setup? I started with docker for simplicity/ubiquity, but I know it's not everyone's cup of tea. The API server in particular can be a bit challenging to set up properly, but if I know what folks are looking for I can try to provide some guidance!

Prologue is also my go to audiobook app, and I really do love it, too. Its a significant inspiration for me for the reader apps; hopefully one day soon it will have feature parity!

Thanks! I think you're probably right. In the meantime, there are some screenshots on the App Store page for the reader app: https://apps.apple.com/us/app/storyteller-reader/id647446772...

I keep forgetting to do this! Here's a link: https://apps.apple.com/us/app/storyteller-reader/id647446772... and I'll push up a change to the docs right now that includes that.

Thanks, I'm so glad folks seem to like it! Agreed, I remain astonished how limited support is for synced narration (forget non-DRM; really only Amazon even provides this feature!). It's totally changed how I consume books. Hopefully EPUB's Media Overlay spec (which Storyteller uses) will become more widespread!

It's on the list! https://gitlab.com/smoores/storyteller/-/issues/9

> I've started listening to audiobooks on my commute, but I read at night.

This is basically exactly why I started down this road over two years ago. I really wanted to be able to switch back and forth between my audiobooks and their text representations!

Thanks for mentioning Storyteller in that discussion, I'll have to hop in there!

Hello! syncabook is awesome, and indeed Storyteller does take "the opposite" approach when it comes to forced alignment.

Others have linked to the docs, where I go into detail about the syncing algorithm, but at a high level:

Storyteller uses Whisper to transcribe the audio to text (this is the most computationally expensive part of the process)

Then we use a Levenshtein-distance-based fuzzy search algorithm to find each chapter in the text (this is attempting to account for the difference between audio and text versions, as you said!)

Then for each chapter, we find the start and end timestamp of each sentence, again using a fuzzy search across the transcription.

In general, Storyteller does a pretty good job; it treats the ebook as the source of truth, which means that at the moment it sometimes misses introductory and ending pieces of the audiobook, though it's on the roadmap to have some support for explicitly triggering those when that happens.

That's correct; it requires the ability to analyze the actual contents of the ebook and audiobook, so they can't be locked down with DRM. This is unfortunately pretty limiting, but at least for audiobooks, online stores like libro.fm have pretty massive catalogues of DRM-free audiobooks!

As others have said, you provide the audiobook (which could technically be something you generated yourself with TTS!) and Storyteller syncs it. However, I've added an issue on GitLab to investigate building TTS directly into Storyteller, because not all books have audiobooks, and it would be cool to fill that gap!