Hacker News Reader

Indeed, this is a deceptively really hard problem that I think nobody perfectly solved yet. The main problem with the deep learning route is it being resource demanding (both computation and memory expensive). Hopefully these problems will automatically go away in a couple of years as the mobile devices become more powerful and the deep learning architecture gets more light-weighted.

Have you tried Office Lens by Microsoft? The scanner part of the app is brilliant

The problem is data. I am not sure how to collect enough data for this easily (on the order of 50k or so, we don't need to train from scratch).

Try the latest update of Genius Scan, it's more like 80% :)