Handwriting to LaTeX maths | Hacker News Reader

Handwriting to LaTeX maths

webdemo.visionobjects.com

652 pointsyannis88 comments

Open articleView on HN

Hi folks,

A small note on what we don't support yet:

no cube roots, no nth roots either

square root's top line must be a single line (make it wide enough up front)

no matrices

no system of equations

no corrections, no scratch out (use top left undo/redo arrows)

leave enough space between integral/summation symbols and main expression for better accuracy

LaTeX output pleases MathJax as much as possible (thank you guys for your lovely rendering library)

Hope that helps, Thank you for trying it out.

PS: MathML seems to get little attention, why is it so?

--- LIST OF SUPPORTED SYMBOLS (encoded in UTF-8) ---

Letters

  a b c d e f g h i j k l m n o p q r s t u v w x y z
  A B C D E F G H I J K L M N O P Q R S T U V W X Y Z

Digits

  0 1 2 3 4 5 6 7 8 9

Maths symbols

  € $ £ ¥ ₩ ¢ ( ) < > [ ] { }  ! # % & ? @ / \ | ∥ © ∂∅ ∇ ∞ 
  ℂ ℕ ℚ ℝ ℤ
  + - ± × ÷ * ∘ · = ' , .  : ; _
  ← ↑ → ↓ ↔ ↕ ↖ ↗ ↘ ↙
  ⇐ ⇑ ⇒ ⇓ ⇔ ⇕
  ∀ ∃ ∄ ∈ ∉ ∋ ∌ ∩ ∪ ⊂ ⊃ ⊄ ⊅
  ∼ ≃ ≠ ≡ ≢ ≤ ≥ ≪ ≫ ∝ ∠
  ∏ ∑ ∫∮∧ √

Greek symbols

  Γ Δ Ω α β γ δ ε η θ λ ν π ρ σ τ φ χ ψ ω ϕ µ

International convention units (with cursive support)

  km hm dam dm cm mm µm
  ha
  hl dal dl cl ml µl
  kg hg dag dg cg mg µg
  ms µs
  GHz MHz kHz Hz

Other mathematical terms (with cursive support)

  sin cos tan sinh cosh tanh arcsin arccos arctan cot coth
  min max arg argmin argmax
  inf sup lim liminf limsup
  ln log
  dx dy dz dt

> PS: MathML seems to get little attention, why is it so?

MathML is hard to write and harder to read. The only time I pay any attention to MathML is when sites do not have TeX as an option. Others may feel differently, but this is my reason.

I somewhat expected people would use it as an input to other math software

I think enough people are used to using LaTeX that MathML was never going to catch on, as its too verbose.

> MathML is hard to write and harder to read.

That's what tooling is for.

It looks like a great solution for including Math in html, only the browser support seems to suck. No Chrome or IE support without the use of a plugin.

> That's what tooling is for.

I disagree. Essentially requiring tools for a supposedly human-readable markup nullifies the whole human-readable aspect. Tools such as equation editors can be a great boon, but those can be had without giving up being able to read and write the markup directly.

I do think it's a shame that browser support for MathML is limited or missing. I'd much rather have direct, native MathML support than nothing. MathJax is a pretty nice way to address this for now, I think. Much better than requiring random site visitors to install a plugin.

I was very impressed that it was able to interpret "min" with "x-2" underneath it, but if I wrote anything next to it the interpreter failed. I tried writing the "min" bit on one side and putting a very large parenthesis on the other side of the sheet, but it still didn't work.

Also, it handles \overline wonderfully -- impressively differentiating between an overline and fraction -- but it has trouble with the \underline. \Underbrace isn't working so well, either.

Yet another difficulty - "1 / 2" is turned into "y_2". The horizontal fraction delimiter doesn't seem to be recognized.

Very impressive!

It doesn't support '⊂'. It supports 'c' and 'C', but not '⊂'.

Edit: It's amazing work by the way, but distinguishing those definitely needs more training.

Writing '⊂' alone would likely be recognized as 'c' or 'C' because the decision is based on the context.

'A ⊂ B' is unkikely going to be recognized as 'A c B' or 'A C B'

is it a raster image recognition, or a vector/gesture recognition?

it's online recognition

we interpret the strokes (1 stroke == 1 sequence of 2D points)

in other words, it doesn't work on images; but we need input from a mouse, stylus, touchscreen, inkling, anoto pen, anything that outputs sequences of x,y coordinates

then it's rather a gesture recognition :)

it would be interesting to see it on a wearable computer, taking input from some gesture input devices: accelerometers or video input with markers

vector and gesture. This demo is a use case of our SDK. The SDK needs in input strokes (array of x/y) in the order. It returns a tree of symbols. We provides tools to transform into LaTex and MathML.

thanks, this is indeed very interesting. I gave a link at a Russian wearable computing community: http://ru-wearable.livejournal.com/107496.html

For some reason, the first thing I tried was circle-plus, which it thought was a theta followed by a bar or a one. D'oh.

Do you have an iOS SDK? What are the licensing terms? I'd love to integrate this into some of my math games.

It works splendidly with any algebra I throw at it, but it's not very much intended to handle expressions in logical calculus, is it?

Case in point: http://i.imgur.com/tnuxs.png

For a bug report like this, you might want to explain exactly what LaTeX you expect that to turn into.

$p,q \vdash p$ — sorry, I was pretty sure it was obvious to anyone, but I guess my handwriting wasn't that good.

Looks like it knows it's supposed to print "sin" and "cos" in non-italic text, but not the 'd' in \int x d x.

Extremely neat in any case.

I've almost never seen an upright 'd' in mathematical literature, though it seems to be somewhat more common in engineering texts. Ditto for the imaginary unit 'i' and the constant 'e'. Matter of taste.

All three are set upright in _The Princeton Companion to Mathematics_ (Gowers).

Hm. Now I wonder where I got the idea that of course it's supposed to be upright. Though I have been reading the Princeton Companion book mentioned in sibling.

I guess the problem is that the system drops the space between the expression and the dx, outputting the "xdx". Math text does have space before dx even if the letter is italic.

Cube roots and nth roots worked just fine for me.

You must be a real handwriting-wizard because I've tried for 10 minutes now without getting it to work! :(

This is the piece we (Khan Academy) need to create a compelling math exercise experience on tablets. Playing around for a few minutes, this product seems to be way ahead of anything else out there.

If you're the creator, should I go through the contact us stuff on your website, or is there a better way of discussing possibilities with you?

(edited: @gpakosz contacted me on twitter)

I saw another app like this (probably here on HN) a while back...it's an open-source Haskell project:

http://detexify.kirelabs.org/classify.html

EDIT: GitHub sources:

https://github.com/kirel/detexify

https://github.com/kirel/detexify-hs-backend

The version of Detexify for the iPhone is friggin' awesome.

I remember once upon a time it was faster for me to write out an equation than to typeset it. Then I met detexify and Mathematica. Both are brilliant, one is free :-)

Hey! I've been daydreaming of making this application for years now. It would be the killer application for tablets (for me, anyways).

I'm an Android developer, so if there's any way I can help out, or if there's any way I can make it happen faster, please let me know! (email in user profile. I'm really really hoping I can help out somehow.)

Cool. Just emailed you.

!FINALLY! I'm very happy with this. The recognition is much better than the other similar things I've tried. By my limited testing, it seems to be accounting for the image as a whole and not the order of strokes, so should work for uploads.

A couple desired improvements:

-Allow uploading of images

-add an eraser/undo

-some Greek/Hebrew letters (aleph, beth, xi... ) aren't being recognized despite my tries... they've very similar to X's, equivilant, and ='s. Beth really should work, but isn't.

There is an 'undo' button at the top-left. Though sometimes it didn't register touches on my iPad.

ah, sure enough. I limit my suggestion to an eraser then.

Their Web Shape demo (http://webdemo.visionobjects.com/shape.html?locale=default#) lets you "scratch out" objects - perhaps they could implement that here? It's what I usually do when writing equations quickly, anyway.

This works surprisingly well! It even made my crappy writing into the correct markup. Great for when you have a big equation to "translate" to LaTeX quickly or to just look up a character you don't know the name of.

I wish there was a function to upload an image or to capture an image via webcam or something like that. Then I could write it on paper and show it to the webcam and get the markup, or upload scans of notes to have them translated.

> or to just look up a character you don't know the name of.

There's Detexify for that:

detexify.kirelabs.org/

Yeah, they could easily create a separate pastebin-like business for equations. It would do miracles for homework assistance and teletutoring. Especially if they let people embed the code.

EDIT: You could even add a business model that made the posts auto-delete after a month where premium users could keep their pastes up forever and embed them. Infinite possibilities.

On my first attempt, I couldn't even make out what I had drawn, but this service inferred my intent anyway. Amazing.

Very impressive! The only issue I've run into is it having trouble discerning lower-case and upper-case letters. Training with my own writing style should be able to correct this however.

Is there a plan to offer this as a non-web-browser service? I would love to be able to write math out on paper, or a resistive screen tablet, and then import it into a LaTeX document.

I am faster writing equations by hand than typing LaTeX (and definitely faster compared to using a WYSIWYG equation program).

Edit: By the way, this is the perfect example of a problem I've always wished a start-up would come along and solve for me.

http://detexify.kirelabs.org/classify.html Found this ages back. It suggests options and not just one result. Works better.

Works only for single symbols, but not for complex formulas.

They are different. Detexify is really useful, since it matches against a vry large symbol set, and is trained against a pretty large sample set. For symbol recognition, this app is pretty crude.

It's still cool, but I struggle to think of a use case for it. It's too unreliable to be of use for scanning maths notes.

Oh. Good point. Yeah Web Equation does better in that regard. But I think its good to give some alternatives as well. Just in case it gets it wrong sometimes.

Seems to work very well. I wonder about the technics they use.

I asked about exactly this a while ago here: http://metaoptimize.com/qa/questions/2097/ocr-lib-for-math-f...

The answers at that point were quite limited. The Tesseract OCR engine was just not made to recognize such structure. There are some other closed solutions, though.

Impressive. It seems to work a lot better than Microsoft's Math Input Panel bundled with Windows 7! Though its not perfect:

http://desmond.imageshack.us/Himg820/scaled.php?server=820&#...

This was interesting. I've recently been using latex to write math equations down but this might save me a lot of time (one I dig my drawing tablet out of the closet).

I'd be more interested in a write-up of how you made it more successful than previous attempts however, technologies used, etc.

So. Awesome. And it works on my phone -- very sweet. My only quibble is that I couldn't get it to hack me up a per mille (this was the #1 question I used to get working as LaTeX support for a journal publisher years back).

I didn't know 'per mil' had a sign! 'Per mil' is one tenth of a percent(%), so 1% = 10‰.

http://en.wikipedia.org/wiki/Per_mil

There's a package, wasysym, which gives you \permil, but I always just cheated with \%_o. Doesn't look very pretty, though. :)

It just works. I've thought it will sometimes work. And it just works.

Brilliant.

That's pretty brilliant. I do wish I could edit the generated LaTeX in place to give it hints when it's gotten something slightly wrong.

I wonder if it has any contextual smarts (e.g., sees a partial derivative sign in a numerator, knows it should "vote up" interpretations of the denominator that start with another partial derivative sign).

It did -\int _{-\infty }^{\infty }\dfrac {\cos \left( \pi t+\beta \right) ^{2}} {e^{t^{2}}}dt=F\left( \beta \right)

I'm impressed.

Awesome! I really hope, someone comes up with a pdf2tex tool, that handles formulas well.

Edit: is even more fun on a tablet.

This works amazingly well. Brilliant !

Just used this on an interactive whiteboard in front of a class for basic algebra. Very nice indeed, well done sir.

How about image save for the rendered formula? I could copy/paste straight into Word for a nice homework exercise.

Basically education lower down the system could use this, not just University level.

Funny glitch: draw an alpha, get \alpha. Then draw a beta and gamma to the right, without touching or overlapping the alpha, and I get \propto \beta \gamma. I've found it shuns greek letters in general. I get an `x' from by best attepmts at \lambda.

Really impressive and useful, nonetheless.

Oh wow. This would make my iPad math game so much better. I already have handwriting, just no handwriting recognition... http://itunes.apple.com/us/app/math-battle/id476344792?mt=8

I always dreamed of having a tablet that could do this when I was taking math in college.

Where's the code?

If you got this to the point where you could scan a document and have it spit out latex code for the whole thing, you would have... ... a huge success on your hand. EVERY mathematician I know, most certainly including myself, would use it constantly. I have pages and pages of disorganized handwritten notes.

Since this is a forum for startup founders, put it this way: such a scanner would be of more use to mathematicians than, say, the sum total of Elsevier's output, and their latest financials show annual operating profit in the ten figures.

FOSS would of course be even cooler (IMHO). But, food for thought... :)

They rely on analysing your strokes. So images cannot be used .

This would be great, however, with one of those smart pen/pad combos that digitally transcribe whatever you're writing/drawing on paper.

Great! But it does not recognise matrixes. Matrixes are the hardest thing to write in LaTeX, it would be wonderful to write them by hand and then having the LaTeX code without problems.

Indeed, we don't support matrices yet but this is a feature we're currently adding, no eta though.

I'd happily pay for an Android note taking app that integrated this.

I'd pay good money for it

Very nice. Always thought the stylus interface had a lot of upsides and this is yet another one.

Looking forward to more devices supporting both fingers and stylus, like the Flyer or the Galaxy Note.

This is pretty cool, although it doesn't seem very efficient if you goal is to write TeX. It would be interesting see how well it works to OCR hand-written notes or old-timey typewritten (that is, typed on a typewriter) math manuscripts where the mathematical symbols written in afterwards by hand...

I'm not sure who do you address with this application. That's how real math looks like: https://nrich.maths.org/discus/messages/117730/117502.jpg

This is extremely cool! I could see using this to generate input for a paper--but a mouse is not the best input device--I'll have to test this on a tablet later--but, even with a mouse, the results were impressive (no hbar yet?)

This is fantastic! It's actually more robust than I thought it was going to be.

I just noticed that they even have a free app in the App Store: http://itunes.apple.com/us/app/myscript-memo/id446368116.

I had an idea for a mobile app where you'd take a photo of a legal pad full of handwritten text and equations, and the app would generate a latex document or a PDF. Maybe this is possible now?

Try this (free) iPad app. Also contains a numerical solver:

http://itunes.apple.com/us/app/gridpaper/id385633188?mt=8

See also Enventra, works with Excel, Mathematica, and Maple: http://www.enventra.com/

Fantastic work, very excited by this. A "learning" technique would be cool, if I could correct things and the system would learn over time.

This is incredible good.

But some formulas doesn't work (for example: $\sqrt[3]{2}$) and I can't find a feedback / suggests a better translation button.

Was anyone able to get fractional exponents to work? I tried several different ways of writing it and had no success.

Great work! Everyone else is thinking "this + wolfram alpha" == algebra / math analysis problem solver for ipad?

You guys have got to integrate this with teachontablo or something similar. That would be simply awesome.

Is this 100% embedded in javascript? Is the code available un-obfuscated somewhere? thanks!

No. The client requests a server (Jetty). The server uses our SDK with the Java version.

the tech is implemented in native code, runs everywhere from tablets (iOS, android) to desktops (windows, mac, linux)

as Mathieu mentioned, the backend of this demo is done in Java which talks to our native code through JNI

Amazing. So much fun, so useful. Will be using this in future homework assignments!

This would have made university so much more fun.

I did not manage to get $\xi$ working.

Very Cool!!!!!

Superwork! Hugs!

"Windows 7 Ink Input and Math Handwriting Recognition"

Nice to see a version on the web. Weird to the web cloning naive software on a few years delay.

Reply on news.ycombinator.com