millimacro

This is very cool! I wonder - does this also work reasonably fast on older hardware?

Cheers everyone for your lovely engagement!! <3

I was wondering, how do you capture handwritten notes using Xournal and Xournal++? Using some sort of tablet maybe?

Great to hear that you found out about Xournal++! It's really the power of open-source that you own your own handwritten notes as you are not locked in.

"HTR is an oasis in this global desert of manual digitization" - hach, this is such a great phrasing! <3

Xournal++ is a great project that features a bunch of really great developers who dedicate a lot of time to it.

I am not much involved in the Xournal++ development itself but then try to utilise my machine learning skills to build an HTR system for Xournal++ in the form of a plugin.

Cheers! :-)

Yes, you're right, stroke-order-aware HWR are hard to find. One reason for that is the lack of good datasets for machine learning model training!

As such, my stroke-order-aware attempt over at https://github.com/PellelNitram/OnlineHTR/ uses a dataset from 2000 with around 12,000 samples. Contrary, the internal Google dataset is reported to feature around 16,000,000 samples :-D.

This is a great observation!

Currently, the machine learning model only supports offline HTR (i.e. using images) but online HTR (i.e. using pen time series data) is in the making, see here:

https://github.com/PellelNitram/OnlineHTR/

Yes, that's totally correct! The current version of the plugin supports only so called "offline" HTR, which operates on images. This is ultimately determined by the underlying machine learning model.

I have developed another model however (based on a somewhat recent Google paper by Carbune et al. 2020), that operates on pen dynamics and thereby implements online HTR, see here:

https://github.com/PellelNitram/OnlineHTR

This model is open-source as well and will be part of the HTR system for Xournal++ in the future. Feel free to give it a try yourself locally.

One question that has been bothering me a long time and prevented online HTR so far for me is how to find text on a page in temporal domain (i.e. in online domain and not offline domain). If you have any ideas on that, please do let me know as I would greatly appreciate that! One possible way is a transformer model - but again that feels a bit overkill and introduces a context length.

I totally agree with you!

The HTR feature in Xournal++ does not require you to scan anything though. You just write handwritten notes in Xournal++ as always and upon saving with the plugin, the resulting PDF is searchable (the handwritten texts at least). So there's no scanning or retyping involved :-).

Yes, that's what I think as well!

Oh that's so exciting to hear! Please do let me know if you need help when trying to use the plugin! I am happy to help you until the plugin works for you.

I decompose my answer in three prime time metrics below and obtain that the plugin is at 6.4/10.

These are the decomposed prime time metrics:

1. The plugin itself is at 10/10; e.g. I use it in production myself.

2. The prediction quality is at, say, 6/10. I am actively working on this.

3. The installation process is 3/10. That's because I have only tested it using my machine setup, for which it actually works fine. To improve here, I'd love to get a bit of user feedback to improve the installation process.

Cheers!