LaTeX, Instantly: Extract LaTeX from PDFs or Handwritten Notes | Hacker News Reader

LaTeX, Instantly: Extract LaTeX from PDFs or Handwritten Notes

mathpix.com

Oh god. An app like that, with a full pdf to LaTeX file converter, would be a godsend for the blind community. We usually sturggle a lot with mathematics in PDF documents, as it's usually not accessible for screen readers at all. TeX isn't the most pleasant thing to listen to, but it's at least plain asci text that any screen reader can process.

well that's an interesting (tough) problem. it's made infinitely more difficult given how many different ways there are to read a typeset expression.

are you blind mathematician? i have a related question: are you an algebraist or a geometer? are you good at mental arithmetic?

Hey, Mathpix founder here. I'm doing some research on how we can do just that, would love to chat if you're up for it, please ping me nico@mathpix.com.

Could you provide AppImage for Linux? Really bored with Snap/snapd.[0]

[0] https://snapcraft.io/mathpix-snipping-tool

This is a 1-liner in Wolfram Programming Language: ‘Speak[SpokenString[ToExpression[texExpression, TeXForm]]]’, for ‘texExpression’ an arbitrary TeX formula.

See https://reference.wolfram.com/language/ref/SpokenString.html

I’ve been looking for a way to convert math texts to audiobooks, but haven’t actually implemented anything.

now that's cool

See https://www.wolframcloud.com/obj/george.w.singer/texToSpoken... for a demonstration. Put any tex string into the form and it will be converted into a spoken formula.

I just posted this in the WCAG Facebook group and found out there’s this as well: https://www.texthelp.com/en-us/products/equatio/

Equatio is powered by MathpixOCR (https://mathpix.com/ocr)

Miki, we take a lot of time to make sure that our multi-level formulas are captured as figures and then alternate text writing out phonetically for our engineering documents. We have been struggling with a better workflow since this take a lot of time when dealing with 1,000 page documents. Our testing criteria are JAWS and NVDA. Any insight you might give on this would be super. Right now it is just a long and tedious process.

I've been looking for an open source solution to take drawn math and convert to text. I don't know if this is it but we certainly need more of it.

Windows comes with one I don't remember the name of, but I'd rather like to have a simple web browser canvas or webgl that could be drawn using a pen on a smartphone.

Last I looked into open source LaTeX OCR, albeit two or three years ago, the landscape was hardly different from this 2010 answer: https://tex.stackexchange.com/a/1476 i.e. pretty sad.

(This is not it because it’s a SaaS product.)

Edit: Also, on the commercial side, I’ve used several MyScript products. They’re decent, but not amazing.

>Windows comes with one I don't remember the name of

math input panel (or integrated into one note). i bought a tablet just for this but it ends up being not good enough to be super useful. i think it's quite a tough task for something like an RNN.

Well, this one is quite interesting, and would integrate nicely with tablet inputs for IDEs, but I have been burned often enough by non-free tools or SAAS (as well as the philosophy/ethics side) that I can't resolve to make closed-source and nonfree software part of my workflow. As a one-off, though? Maybe.

Mathpix Snip is free up to 50 images per month. Also, we are open sourcing our JS rendering stack very soon, which we hope will make publishing scientific content online more competitive with sharing PDF links.

FWIW, totally agreed with @MayeulC's comment (to the folks who might be willing to pay for this, the content is too damn important to risk platform lock-in. For this service to even get consideration, I would imagine that everything must be cleanly exportable, such that one needs only pay for continued use of the service).

That said, where I would seek a tool that really adds value is auto-converting notes (text and math interspersed) into LaTeX, so that (as a value add) it can also be intelligently searched (coupled with an indexing+search solution). I think many academics/knowledge-workers might be open to paying for such a service.

Bonus if you can work well off just smartphone photographs of pages.

The content generated from Mathpix Snip can be pasted into any Latex compatible editor, so that avoids the lock-in you sometimes get with proprietary WYSIWYG editors. Plus as I mentioned we'll be open sourcing our rendering engine soon.

Thanks for the suggestion about adding search capabilities, we're thinking about this.

If you curious about it, I would suggest to just sign up and try it. If you spend a lot of time typing up Latex it's definitely worth having, especially for matrices / aligned equations.

The API pricing with 1000 free calls per month seems way more lenient than the free plan for end users. Are you worried about end users writing a simple script or downloading someone else’s simple script to sidestep your intended free quota?

why do you ask for organization access with github oauth?

> non-free tools or SAAS (as well as the philosophy/ethics side)

The philosophy half I get, but ethics? Any chance you could explain why paying for tools or services is unethical?

They recently implemented a pretty restrictive freemium subscription model that has tempered my enthusiasm for it massively. 50 snips a month is nothing for grad students.

Do you think 100 free Snips per month would be better? We're thinking about how to do a student plan.

If grad students in relevant fields aren't one of the main target markets for this kind of product, where do you think they're going to make their money?

Grad students will just just maple/mathematica to directly generate latex from an expression.

Which grad students? In CS the use of these is long gone (at least in the two departments I’ve been in). I wonder how prevalent they still are in physics.

But I’m pretty damn sure that physicists, mathematicians and computer scientists all just type their equations straight into latex unless it’s a super complicated computer generated formula, which you don’t typically write in papers..

How are they supposed to pay themselves in that case? It’s an incredible service.

4.99/month for unlimited snips isn't exactly onerous. Grad students are known to be able to afford Netflix and Spotify.

I used to think converting from handwriting to LaTeX would be a really useful program and save a lot of time, but after typing LaTeX on a regular basis, the input speed is really not an issue. Sometimes I'll encounter a symbol I don't know, and then I draw it in here [1] and remember it for next time. When doing math, it's definitely not the bottleneck in getting my work done. I would definitely be slower with handwriting conversion, especially when you factor in fixing errors in its translation. I only see handwriting conversion being useful as training wheels for beginners, but would be interested if someone has a different experience.

Automatic recognition from PDFs is more promising for accessibility and reformatting, and also seems to be easier to do since typeset math is easier to recognize than handwriting.

[1] http://detexify.kirelabs.org/classify.html

If you’re not at a keyboard, or prefer to use a stylus, typing really isn’t an option. Or if you want to capture the actual working-through of a problem, rather than working from an already-completed work. There are a number of use cases you might not find useful with respect to handwriting, but others might find incredibly practical.

I am comfortable enough with latex that I often work through problems on latex. For certain cases, with long expressions, it is actually simpler than handwriting, because I can copy an equation onto the next line and edit it in-place, hence, taking less time and ensuring I am not missing terms.

I have a friend who got so into latex that a few times he would mindlessly use latex code while handwriting things, like writing "\alpha" on the board instead of drawing the greek alphabet.

Just curious if there is any plan for the Mathpix OCR/engine to transcribe tablet notes in real-time to a LaTeX document? That would be the holy grail for me.

Yes we're looking at this. What type of tablet do you own?

Currently I'm not using a tablet. But, I'm doing an M.S. in Actuarial Science. With > 90% of my notes being formulas, I would buy a tablet just for this feature :)

Remarkable?

Android on Boox note pro

This could blow Nebo out of the water.

I actually worked on transcribing math expressions to latex a few months ago with a Resnet backbone + LSTM + Attention.

All my code to train the model is available On GitHub: https://github.com/bkkaggle/math-recognition

If anyone wants to chat about math expression OCR, I'd love to talk about it!

It's a surprisingly difficult and interesting (and under appreciated and unknown) area of research!

This doesn't seem to support Wayland? Trying to create a snippet corrupts my whole screen. If I can get it to work, it seems quite useful.

This is what I get when trying to capture an image: https://cdn.mathpix.com/snip/images/OvK0viJ9Q4k4evoCpnfvtIFl...

Hi, no we do not support Wayland. You can, however, try pasting images directly into our notes app snip.mathpix.com by copying the image to your clipboard and then pasting directly into the document.

I used this recently as part of a literature review I was doing. Worked amazingly well and sped my work up substantially. Highly recommended.

I enjoyed the Isaac Newton "testimonial" (the others as well, but that one was the first I read).

@nicodjimenez Could you give the HN audience a quick idea of the technology behind this? Recurrent ConvNets like in https://guillaumegenthial.github.io/image-to-latex.html ? I remember seeing an "image to LaTeX" academic challenge, do you plan competing in it, if it still exists?

FYI, I'm pretty sure this is wrong:

    \ell ( \theta ) = \sum _ { i = 1} ^ { m } \operatorname{log} p ( x ; \theta )

Given that we're summing over i = 1 to m (in the picture), I think that's supposed to be x_i :) Impressive, nonetheless.

Just one note: Ctrl+Alt+M is a horrible choice of shortcut, as it conflicts with typing characters on many keyboard layouts. μ on my keyboard, for example. Ctrl+Alt is identical to AltGr and thus should be avoided for shortcuts in general.

You can record the shortcut to be whatever you'd like in your settings :)

Has anyone on this thread actually used this?

Would be interested in a product review

Is there a recommended cloud or app solution that works well on the iPhone/iPad for writing a lot of LaTex?

There are several LaTex environments for iOS. I habe been using TexWriter, but very lightly so far. It worked well for me.

I wonder if it could be (ab)used to LaTeX-ize plain TeX.

there is also https://github.com/falvaro/seshat

i've used this. it's a really good solution to an odd problem. how often do you need the latex from a paper or book or whatever instead of a screen cap? if you're going to fold it into your own writing then you'll just write it out yourself since most stuff isn't that complicated (and because ostensibly you know latex already). if you're not going to fold it in (and just use it in a powerpoint or something) then max-zoom+copypaste works fine.

on the other hand if it could do tikz now that would be magical (but as well know that's much much harder).

Reply on news.ycombinator.com