321 karma · joined February 3, 2023
The current Storyteller alignment algorithm actually does do just that! We use Whisper to transcribe the audio to text, and then use error-align[1] to align on the text.
There are a few disadvantages to this approach:
1. Whisper only supports ~25 languages, and only about 10 of those very well. We want to support more languages, and Massively Multilingual Speech supports "1000+" 2. Whisper's timing outputs are not very good. We want to do word-level highlighting, like in the demo at the top of the post, but in order for that to be a good user experience, those timings need to be very precise. Much easier to do that with CTC!
CTC Viterbi is the tried and true forced alignment algorithm for good reason. It's not really that it's heavier duty than running Whisper and aligning on the output. Rather, it's like you stop Whisper early, before it does the final step of actually producing text, and step in and say: take the data you just calculated and use it to produce _this_ text, specifically. And then, since it produced _your_ text, you don't have to do anything else, you just use the timestamps directly.
The only reason Storyteller never used it in the past is because I couldn't come up with a good way to do the boundary search I describe in this post! This is super important for books in a way that it may not be for your oral reading transcript use case, because chapters can be (and often are) out of order between the ebook and audiobook. But once I worked out the n-gram RANSAC approach, it became much more tenable.
Also, disability isn't binary? Some people have an easier time reading text than others — is your argument that books should remain less accessible to those people for whom it's harder because you don't personally feel like they should need it?
And then there are _lots_ of people that just enjoy reading this way. Audiobook production is its own art form, and many people like experiencing the text and the audio together. For some people it helps them focus in a noisy environment, for some people they find they can read faster with the narration, and some people just enjoy it.
Every single time I post about Storyteller, there are multiple people in the comments insisting that this shouldn't exist because no one should ever consume books differently from how they do. I don't understand it, honestly. If you don't want to read books this way, that's totally fine! You can even still use Storyteller — it works great with plain EPUBs. Why the need to make others feel bad for the way they engage with stories? The world is a better place if more people read more books — discouraging people from reading books in the way that feels pleasant and accessible to them makes the world worse.
I think if you feel compelled to instruct people you don't know to "stop their current habits" because you don't personally understand their needs or wants, perhaps you should reconsider your current habits, yourself.
When we do roll out multi-level granularity, it will indeed be something that you can configure yourself, including how each level is indicated (e.g., you might want to set a background color on the sentence and underling the word) and whether a each level is indicated at all!
The "spread out" highlighting is a really neat idea, I don't think I've heard that suggestion before!
"Immersive reading" seems to be the industry term for this feature — I used it here because I though it was most likely to be recognized by a wide audience. Personally, and within the Storyteller ecosystem, I call it "readaloud," which I think is at least a little bit more useful of a phrase.
I supposed I should add: it's also, obviously, fine if this just isn't for you! Lots of people get a lot of value out of audiobooks (e.g., I really like to listen to audiobooks in the car or while running), but like to switch to reading when it's an option. And lots of people find readaloud super valuable, whether because they have a print disability or neurodivergence that makes reading challenging, or just because they like the experience of being read to. But lots of people are in neither of those groups!
1. StorytellerSync (https://github.com/Sirozha1337/storytellersync.koplugin), which syncs your KOReader progress directly to your Storyteller server
2. Audiobook (https://github.com/stradichenko/audiobook.koplugin), which has a WIP media overlay implementation that works with Storyteller readalouds
The Storyteller mobile apps have great support for this, I think. They have fully fledged audiobook players and ebook readers, and you can switch between the two with one tap. And then of course you can also double tap on a sentence and start playing from there, with the app highlighting the currently read sentence.
Anyway, I am really pleased with how the new algorithm turned out! Hopefully someone else finds it interesting as well.
Handle with Care is a software collective that builds and maintains open source rich text editing libraries, including React ProseMirror [1]. We all came from The New York Times’ content management system team, and we spend a lot of time thinking about rich text and collaborative editing.
Now we’re working on something new: Pitter Patter will be a fully featured collaborative rich text editing toolkit, with all of the bells and whistles you need for your own text editor.
The space we’re entering is not devoid of solutions — Lexical, Slate, ProseMirror, and Tiptap are all viable options for building modern, browser-based rich text editors. But we feel pretty confident that we’re going to be able to bring some value, nonetheless.
First of all, Lexical, Slate, and ProseMirror (especially ProseMirror, in our opinion!) are all excellent rich text libraries, but they are also quite low level. You can build nearly anything atop them, but you will have to do quite a lot of the building yourself. Sometimes that’s exactly what you’re looking for — in that case, Pitter Patter can still provide you some value, because we’re going to be releasing individual libraries (like a CodeBlock node view, advanced markdown serialization, and suggest changes) that interop with the existing ProseMirror ecosystem.
But if you want something that’s more batteries-included, you’re mostly left with Tiptap. Tiptap has been dominant in the space for a while, but we think we can do better!
- We’re building on top of React ProseMirror, a truly React-native ProseMirror view, that doesn’t have to make any of the compromises that Tiptap’s React integration currently makes [2]
- We have a deep understanding of ProseMirror’s internals (and we’re not afraid to use it!)
- Pitter Patter will be completely open source
- We’re building on top of prosemirror-collab-commit, the best (only?) rich text collaboration protocol that is both correct and fast [3]
Anyway, we’re posting here for two reasons:
1. Maybe there are some more collaborative rich text editing nerds here that will be exciting (or not!) to hear about this. Sign up for our newsletter if you want updates!
2. Maybe there are some companies that are looking for alternative solutions to what’s out there. Consider sponsoring us on GitHub [4], or reaching out if you want to be more involved!
[1]: https://github.com/handlewithcarecollective/react-prosemirro...
[2]: https://smoores.dev/post/why_i_rebuilt_prosemirror_view/
While migrating to React ProseMirror, initially I tried out just passing the primary editor's EditorState directly to the preview editor's <ProseMirror /> component, but the top level node view components turned out to be just slow enough to render that rendering them twice on every keypress introduced a noticeable lag. So I added a useDeferredValue to render the preview editor in a Transition! Here's a post about how that works and the tradeoffs involved. I added some interactive demos to illustrate how the Transition changes the render flow.
A challenge I foresee is that the media overlays are only reusable if you have the exact same input EPUB file, and have processed it with Storyteller to mark up the sentence boundaries. EPUBs have unique identifiers, though, so maybe this would be fine! We’d need to add a new processing flow to Storyteller, but it should be doable.
Feel free to hit me up in the Storyteller chat if you want to discuss more! Thanks for sharing this idea!