A return to hand-written notes by learning to read and write | Hacker News Reader

A return to hand-written notes by learning to read and write

research.google

595 pointsmfiguiere181 comments

I recently purchased a small refrigerator whiteboard and it's been really amazing with the combination of my iPhone's ability to take a picture of my handwriting (script or cursive) and copy/pasta into a text. It's not always perfect (nor is my handwriting!) but it's good enough to just replace a character or two and hit send.

This really tickles a bunch of things for me:

1) I am not sending a whole image (it's efficient)

2) I don't have to type/swipe at all (I'm not looking at a screen)

3) My s/o has easy access to the list at all times (it's not in a cloud)

4) It requires no power to update/maintain (markers last a long time)

5) It just feels so natural to grab a marker and write on the fridge when I exhaust something in that same fridge.

This is a nifty idea. I bet there'll be a HN-correlated pop in fridge whiteboard sales from this.

Here's a suggestion if you want to avoid markers and such: LCD drawing tablets, I'm talking about the ones that look and function similarly to Etch and Sketch, but it uses a stylus and an LCD display, you can get these for less than 10€.

E-paper tablets these days run android, have fast refresh rates and multiple colors, they're great.

This is very cool. Here is interesting application of something like this. My handwriting is pretty bad, and worse still when writing fast. When I am teaching, a lot of what I write is worse than I would like it to be.

I could teach a system like this my very slow neat handwriting. And then as I write on my whiteboard while teaching, it replaces my quick bad handwriting with the neater handwriting.

> My handwriting is pretty bad

i know it might sound dumb, but have you tried playing with a fountain pen?

The feedback is way different from a ballpoint pen and it also depends on paper and the kind of ink. It makes writing way less "predictable" and a bit more enjoyable.

a cheap one (5-15$) with a medium nib might be a good start... some people move on to collect fountain pens, but i do most on my (on paper) writing with a ~20$ Pelikan Jazz.

I know your comment is in earnest and I don't mean to make fun of you, but there is something so funny in our Americanised world where everything is reduced to "it's not you, you just need to buy the correct gizmo that will solve all your problems."

Fountain pens enthusiasts are like music gear or mechanical keyboard enthusiasts, that justify their hobby and believe spending on the next shiny thing is the key to fulfil their whatever, until the next shiny thing arrives.

The thing is... if one dislikes or doesn't care about writing that they have basically forgotten how to, it is not spending money on a fancy writing implement that is gonna turn them into a medieval monk scribe.

> but there is something so funny in our Americanised world where everything is reduced to "it's not you, you just need to buy the correct gizmo that will solve all your problems."

OTOH, it's an improvement over the other world of solving everything through "discipline" and other kinds of wishful thinking.

Like, you can complain and worry that your kid can't seem to learn how to cut things right with their scissors, try to force some discipline and conscientiousness into them - or you can stop causing them and yourself so much grief and realize that a left-handed person needs left-handed scissors, as using the wrong type for your hand works to prevent the cutting action.

> OTOH, it's an improvement over the other world of solving everything through "discipline" and other kinds of wishful thinking.

kinda, the truth is somewhere in between in my opinion.

most things related to our body movement do require training to be mastered.

I think that writing is no different endeavur. It's just that fountain pens can make it more pleasant.

> The thing is... if one dislikes or doesn't care about writing that they have basically forgotten how to, it is not spending money on a fancy writing implement that is gonna turn them into a medieval monk scribe.

The thing is, it does actually... Because it "forces" you to slow down and take time for it.

I hated writing (heavily dyslexic) but after getting a fountainpen and purposely slowing down I noticed it went better. Now I write in my own language again, I spend whole evenings writing scenes, essays, debates and letters.

Sometimes it is the tool that forces a bit of change (if you want to change of course).

A LAMY Safari with its triangle grip vastly improved my handwriting by forcing me to use a different grip than I’d naturally use. So yes, for me, the correct cheap gizmo solved all my problems.

I've played with different miracle gizmos all my life, nothing worked. When I go slow and careful I can write like a 2nd grader. As a kid my teachers were constantly mad at me until one realized I was trying and got me testing - sadly dysgraphia (likely the diagnosis I needed, though I was never formally diagnosed) wouldn't exist for several more years and so I couldn't get the right help. (if any help exists, I haven't been able to find anything useful and I'm not sure as an adult if it is worth the time - there are so many other things I can do instead)

To be clear, the Safari showed me that I can write without loathing the process. I feel I've given it a fair shot, though, and I'm back to typing everything instead. I'll use a paper and a pen to jot quick notes during a meeting or something but that's the extent of it.

Turns out I can get by just fine with hardly ever handwriting anything. The only people disappointed by this are my elementary school teachers who had insisted this was something I needed to care about.

> Fountain pens enthusiasts are like music gear or mechanical keyboard enthusiasts

There is a subreddit /r/mechanicalheadpens for these three specific interests.

I get where you're coming from, but, given that I own several pens, and my handwriting really differs depending on which pen I'm using, I would say he has a point. Changing the pen you use (even within ballpoints) can significantly impact your handwriting.

In reality, it's a bit more complicated. I've found the following impacts my handwriting:

- Grip format on pen (i.e. shape of pen)

- Ink I'm using

- Paper I'm writing on

- Nib

- And yes, type of pen (ballpoint vs fountain pen)

Having a good pen really helps a lot with writing. I'm not a fountain pen enthusiast, I don't even have one. I just noticed that there's a huge difference between a ballpoint pen and a "rolling ballpoint pen". The naming is confusing, since the ballpoint pen should also be "rolling", but whatever.

Ballpoint pens use a viscous ink that, like graphite in a pencil, needs pressure to be applied. Rollerballs and fountain pens both use low-viscosity inks that flow simply from being touched to the paper.

The latter requires much less effort to write (no constant pressure) and enables writing with the hand held mostly still, using the larger muscles of the upper arm and shoulder to create the letters. The downside is that they can create impressively large ink blots on your clothing if uncapped/unretracted, and the ink can be smeared if you touch it while wet. But pretty much every writing system still out there in use (i.e., not cuneiform or runes) was designed with a quill or brush as the instrument.

> But pretty much every writing system still out there in use (i.e., not cuneiform or runes) was designed with a quill or brush as the instrument.

That's if you know cursive. I don't think the standard small-case script most people use in their everyday life is quill-friendly.

Your style might have to change a bit, but disconnected letters were quite common in medieval Roman-style scripts, which were definitely written with quills.

> that flow simply from being touched to the paper

Or just by itself if you bring it with you on a plane :)

This looks like a perfect rabbit hole I'd be wise to avoid. At least I have a good excuse of being left handed since I would be constantly smearing all the wet ink.

Although it's not a beginner fountain pen in terms of cost (though it is not too expensive, basic models around $160), the Pilot/Namiki Vanishing Point is a retractable fountain pen that does not leak when retracted.

The cartridges for it can, of course, get expensive, but a 1 mL syringe and a big bottle of ink (even Mont Blanc ink is only $25 for 60 mL) will let you refill them cheaply.

I made the original suggestion explicitly to avoid the rabbitholes.

I use a Pelikan Jazz (20$) and a bucket of black cartidriges (100 pieces) I got off amazon for like 8$ (like two years ago).

I got a kaweco fountain pen a few months ago and i honestly regret spending those money, it's a shitty pen, some of the most dumbly-wasted money of my life.

Didn't intend to drag you down into one. Sorry.

I got one because my wife, who actually is an aficionado, suggested that it would be a good one to carry with me (and it is) due to the non-leak. I would carry it more except that we have some forms at work that are duplicates, requiring pressure, so it's useless for that. A Pilot Precise V5 RT retractable rollerball is nearly as good and cheap (in small quantities, $2-3 apiece).

It would be more charitable to say that the change forced upon yourself by changing your instrument is a straightforward way of making you mindful of what you're doing and make it easier to break bad habits. Yes, you can break bad habits by not doing them with the same tools, but the objective is to change your behaviour, not to demonstrate personal Calvinism.

I definitely write better with some pens than with others, and probably best with fountain pens (I have lost my best fountain pen though).

Its not going to make me write like a scribe creating an illuminated manuscript, but there is an awful lot of room for "better" between that and my usual handwriting with a cheap ball point.

> but there is something so funny in our Americanised world where everything is reduced to "it's not you, you just need to buy the correct gizmo that will solve all your problems."

Sigh... I'm not american and i don't live in the US.

> Fountain pens enthusiasts are like music gear or mechanical keyboard enthusiasts, that justify their hobby and believe spending on the next shiny thing is the key to fulfil their whatever, until the next shiny thing arrives.

Yep, I'm aware, that's why i was explicit on the fact that a ~20$ Pelikan Jazz is just great and "getting into fountain pens" is something that you can definitely avoid (I do avoid it, as a matter of fact).

you do you i guess. good luck.

The recommendation was to try a $5-15 pen.

That's a long way from what you're describing.

It's exactly what millions of schoolchildren are required to use when they're learning to write.

Sure, but billions of people can write just fine with a $0.50 ballpoint pen.

Also, it's not $25 or $2,500 that will stop you from doing 95% of your writing on a physical and smartphone keyboard because that's how society works nowadays. This comment cannot be written with a fountain pen, nor my work emails or communication with friends and relatives. Many people can't write any more simply because they don't really have to.

Probably good advice if you are right handed and have good fine motor skills.

At least way into the 90s kids here learned to write with fountain pens. For me this meant pages full of smudges, forked pens and generally unreadable text.

As soon as I was allowed to switch to a regular pen my handwriting improved a lot (still not great, but better).

I think you don't even need to go to fountain pens. As a long time user of ballpoint pens, I recently started using gel pens and I wouldn't go back.

I still prefer using ballpoints with schmidt easyflow 9000 ink[1], but yeah rollerballs (gel) are a great midpoint between fountains and ballpoints. My Zebra G-750[2] is super smooth.

[1] https://www.jetpens.com/Schmidt-EasyFlow-9000-Hybrid-Ballpoi...

[2] https://www.jetpens.com/Zebra-G-750-Gel-Pen-0.7-mm-Black-Ink...

You started a flame war. So let me just say that I exclusively use a fountain pen to write. That handwriting is a lot better. But when writing digitally for teaching or meetings, I use a wacom tablet. That is pretty bad for writing.

I've realized that when I use cheap pens on hotel stationery, my handwriting looks terrible, probably because the surfaces are too smooth? Other than fountain pens, are there other alternatives that give more tactile feedback?

Nibs / dip-pens ;-)

It's only half a joke: having to regularly dip the pen in ink, be mindful of how much ink you have, having to swiftly wipe it once in a while to avoid drying ink (& flow issues), forces to slow down & take "micro-breaks".

This benefits the handwriting, but also the quality of the study. And is surprisingly relaxing.

(so called "crow-quill" nibs are relatively cheap, available, carry a fair amount of ink)

What you are asking for is more "tooth", something mostly determined by the paper. Most stationery fans prefer smoother papers, but I agree that after a certain point, increased smoothness makes my handwriting worse.

If you are stuck using a very smooth paper, I would suggest either a fiber-tip or a drier gel pen. Gel pens with a clicker (e.g. Zebra Sarasa) tend to write drier than those with a cap (e.g. Uniball Signo). Also try a fatter tip. Although this may not be acceptable in many situations, a soft pencil may provide even better control on smooth paper.

yes, that's what I've noticed; when I write in pencil, my handwriting almost looks good (!), but when I try to do the same with a Bic, it's terrible.

Will give the Zebra Sarasa a try.

IMO felt tip pens feel comparatively rough to gel, and ballpoint pen, etc.

I write a lot with a fountain pen and if anything it made it harder to understand my handwriting... even the next day I'll have a hard time.

I've considered the "just learn to write better" approach and I've tried here and there but I've been handwriting journals for 28 years and I'm just not sure it's possible to write cleanly at the speed I handwrite at this point. Especially since it's in cursive.

I can't imagine anyone writing at their 'fastest' is going to produce something that is broadly legible.

Slow means smooth, smooth means fast.

Improving your hand writing is not hard. For whiteboards start out with using block letters only. It will slow you down in the beginning but not for long.

That's one of the "game changing" hints I received during my time as a tutor at university. (One other was to always copy books from back to front; very useful but somewhat outdated now.)

>One other was to always copy books from back to front; very useful but somewhat outdated now.

What's so useful about that?

The photocopies come out stacked in the right order maybe?

This would be a kind of interesting data structure. A stack, FILO, but you can flip it for reading (maybe flip is an expensive operation).

Grass is good.

Slows you down if I had to guess

If your copies come out of the machine right-side up and they get stacked on top of each other, then you can just take the finished stack from the machine when finished. Otherwise, you’d have to reverse the stack, which in real life or in a computer is an expensive operation.

I thought we were talking about handwriting practice haha

My mother had an inkjet printer where the pages came out right-side up. Multi-page documents finished up in reverse order. It was an infuriatingly awful piece of design.

Others already said it, but yeah, the copies came out ready-to-read. The university had, IIRC, Sharp copiers which put out the copy right-side up.

> For whiteboards start out with using block letters only. It will slow you down in the beginning but not for long.

I strongly dislike this advice. The slowing down is probably actually the feature here that works; as for the uppercase, it definitely hurts comprehension. Go lowercase, slow, and deliberately never joining any letters, for your beginning, and I think it will probably be better for all parties involved: reading and comprehension will be improved, you'll get at least as much improvement in your handwriting, probably more, and certainly in a more useful area.

You are right, it is mostly about the slowing down. Switching to the unfamiliar block lettering forces you to go slow. It's a trick, so that you do not have to be deliberate about writing speed but about lettering. In my experience the latter is easier than the former, but obviously YMMV.

I don't know if uppercase hurts comprehension. I think, I have no problems whatsoever. But block letters are not all upper case anyway. You make them of different size to signal upper and lower case, i.e. https://en.wikipedia.org/wiki/Block_letters

Or you could just find some lettering manuals and improve your handwriting. Practicing at a slow speed will improve your fast work, too.

If you draw your equations well enough, they can get converted into LaTex in realtime and then you could run them in a computational notebook.

Esp if you fuse the audio of you explaining the equations along with the LaTex it can correct for errors.

What software can do this?

I think Mathpix API [1] can use used to do something like that in realtime/ish

[1] https://mathpix.com/

The new Calculator by Apple is supposed to do it (but the result is quite underwhelming)

Why not simply have a laser projector, keyboard and canvas textbox then?

From a quick visit to his profile (linked website), he is a physicist. This technology setup is very complicated and against the eternal usage of blackboard in a typical physics department. And to be honest this applies to his suggestion as well but you still at least get the feeling on writing on a board.

Of course, all true physics happens on the blackboard, in a notebook (or in Mathematica).

But, I am forced to use digital tools occasionally, and I am not opposed to improving them.

You also need the skill of lightning-fast LaTeX typing, and the skill of drawing and drafting with a speed comparable to that of a chalk. You need a canvas-driven tool for that, and your eyes would be on the screen for long periods of time, not contacting the audience.

There are regular white/blackboards that you write on with regular marker that just has a tracker on it, so content appears on the canvas on screen for those who are remote. More advanced versions also have laser projector that can project animations, moving diagrams and text on the same board. My suggestion is to tap the board on the place he wants to write and just type it on regular keyboard, hardly a distraction!

apple notes on new iPadOS does this right now -- just cleans up your handwriting to be slightly neater but still look like you.

That's actually a brilliant feature! Without losing the "you"

Wouldn't it make even more sense to replace your quick bad handwriting with perfect Helvetica?

I got a stylus for my iPad and i felt that way at the beginning but I learned that i visualize in my head what I'm going to create right before I draw, and the difference between what's in my mind vs what gets produced is so great, that it feels like a weird uncanny valley and i hate the output.

Not necessarily - your handwriting can sometimes be subtly optimized to the sort of subject matter you write about. For example, when I write lowercase T, I give it a little hook at the bottom. I also write my lowercase L in the cursive form, the tall skinny loop. That keeps similar-looking letters that I use a lot distinct. There must be other examples.

In my handwriting, the letter x is two crossed lines; the algebraic variable x is two Cs back-to-back. This makes it more distinct from a multiplication sign.

There is (subjective) charm in handwritten text that is simply not present in digital fonts. I would rather keep that magic alive.

I think handwriting is a very personal trait. Some people value good handwriting and they write neat even when they write fast. Others don't. Sadly, a vast majority if the world exists in the other camp. This is why they invented typewriters. If everyone conformed to good handwriting and could agree upon a good writing style standard, the world would have been very different.

I like good handwriting, but good, fast handwriting takes thousands of hours of practice. We used to spend a good chunk of school drilling it into kids, but now it's really hard to justify everyone spend that kind of time when we can technology our way around it and there are other valuable skills to learn.

That’s the same argument for using calculators in the past 30 years and ChatGPT in the last few years. At some point, we lose far more than we gain by “technology our way around” our early development.

Why bother having campfires when we have portable space heaters? Heck, why bother camping at all if we have a nice comfortable space at home? More generally, why do any of the things that connect us to our past?

Personally, I think it’s important to learn by doing and then provide a “here is how we made that easier, and now you know why” type of foundation. Perhaps better is having people develop versions of those solutions for themselves so we don’t just expect someone else to solve all of our problems.

If you live in an urbanised country you don't build a lot of campfires- besides there are rules on air quality that prohibit them.

I've known people who valued good handwriting but weren't particularly capable of writing neatly, and others who didn't particularly value it but were capable of staying neat while writing quickly. I think you might be overestimating how much of it is tied to what people value.

> This is why they invented typewriters.

Historically, that didn't seem to be the main reason for the many inventions of the typewriter. See https://en.wikipedia.org/wiki/Typewriter#History

That wikipedia article was a great read but I have to agree with GP.

> According to the standards taught in secretarial schools in the mid-20th century, a business letter was supposed to have no mistakes and no visible corrections.

Certainly, there were other reasons like speed, repeatability, official-looking, etc... too. Most typewriters wanted to be cheaper and on-demand printing presses. Some even managed variable width fonts! :)

Oh, the original comment is certainly right in spirit, I just wanted to be pedantic.

But I'm not sure why you quote something about the md-20th century, when we are talking about the (many) invention(s) of the typewriter? They were already old and well-established technology at that point in time.

When I'm taking notes, I use arrows, squiggly lines, symbols, I circle text, etc. which won't necessarily translate to typical text block.

Hard to make boxes and arrows out of helvetica, and basically impossible to do it quickly

Fine, make it Helvetica plus whatever other shapes you need. I would assume that Helvetica includes most of extended unicode, since it's such a widely used font, but I could be wrong, and it's beside the point.

If the computer is the one transforming your writing it should be perfectly quick.

To be clear, afaik current handwriting recognition software is not good enough for this. But if we had software that could transform bad handwriting into good handwriting, why not go all the way?

Cuz quality is not equivalent to standardization, and sometimes the latter is undesirable

From the title, I naively assumed this article would be about people relearning to make legible/beautiful handwritten notes after losing this ability. That is something I’m currently struggling with after many years of too much typing and not as much handwriting.

Google’s actual research does help people like me, by making our notes less awful digitally. But I’d love not to be dependent on tech innovations to make my handwriting better.

If you’re serious about this, I’ve stumbled into areas of YouTube with people dedicated to this. Pick a font of how you want your writing to look, and practice, practice, practice. People make available (and sell) special lined sheets to get the height of various things right or help to guide the writer to the perfect slant.

You just have to have the time and interest to do the work, much like you probably did when first learning to write.

How do you recommend that practice to be like? Simply trying over and over to perfect each letterform on empty lined sheets, based on a reference?

I’ve done a few handwriting workbooks, mostly consisting of block letter templates to fill. But each of them is incomplete on its own, and they don’t even share the same font, making it much harder to practice.

I'm also curious about the special sheets. Maybe I would be able to write legibly for once in my life :D

Good idea. May you please share youtube link you are referencing?

Also check out the resources on the r/handwriting subreddit.

This has been mentioned somewhere else in the comment thread, but if you want to improve your handwriting, a good way is to use a fountain pen.

My handwriting is immensely better with a fountain pen than with ballpoint or gel pens; I suppose partly because the fountain pen forces you to an optimal position and angle (it's much more inflexible about that, you can't just push it against the paper in any angle and expect it to write), and partly because it provides a smoother experience and feedback.

You don't need to go overboard, the typical €20-ish Pilot Metro with medium nib or similar is more than enough.

Study comic lettering. Not saying it's the most efficient way of writing, but the process will teach you to think in terms of strokes and consistency. You can easily develop your own style from there.

I noticed my handwriting was terrible, and consciously improved it by writing slower and being more mindful of writing neatly. A fountain pen helped me slow down, but fundamentally it was just a matter of slowing down and consciously forming nicer characters until it became easy to do so, at which point my speed increased--but I retained the habit of paying enough attention to make nice characters.

Deliberate practice (as in worksheets or exercises) is less important than just going as slow as you need to, to make the characters correctly, until your muscle memory builds up and brings back the speed.

I suggested Kate Gladstone's Handwriting Repair site elsethread:

https://handwritingrepair.info/

or see:

https://sites.google.com/view/briem/free-books

or John Howard Benson's lovely _The First Writing Book: Arrighi's Operina_ or Carolyn Knudsen's lovely _An Italic Calligraphy Handbook_ (which is much better than the modest title implies) and get a chisel edge marker or fountain pen.

I tried to use tesseract for OCR, 10 years ago, it recognized English good enough. tesseract was also developed by Google if I am not mistaken, but open source.

I tried to use it then, for non English language, for Greek, and it was very bad.

Happy to see some good OCR research based on transformers.

I’ve been really impressed with Tesseract - I used it last month to add invisible OCR text (1) to scanned PDFs I reference a lot. My scans are quite good, but I was still impressed with the accuracy.

I also OCRed the TOC, playing with the page segmentation setting (2) in the terminal until I got output I could copy & paste to add a navigable table of contents.

1: with the help of https://github.com/ocrmypdf/OCRmyPDF

2: https://tesseract-ocr.github.io/tessdoc/Command-Line-Usage.h..., “ Using different Page Segmentation Modes”

I OCRed your comment with Tesseract:

``` I've been really impressed with Tesseract - | used it last month to add invisible OCR text (1) to scanned PDFs I reference a lot. My scans are quite good, but | was still impressed with the accuracy.

| also OCRed the TOC, playing with the page segmentation setting (2) in the terminal until | got output I could copy & paste to add a navigable table of contents.

1: with the help of https://github.com/ocrmypdfiOCRmyPDE

2: https://tesseract-ocr.github.io/tessdoc/Command-Line-Usage.h..., “ Using different Page Segmentation Modes” ```

This kind of mirrors my earlier experience with Tesseract, if it can't get OCRing a screenshot right, what can it get right? It's not like "I used" is such a rare phrase either, but it replaced the I with a pipe.

>if it can't get OCRing a screenshot right, what can it get right?

Book scans which is what it was designed for.

If you read the fine manual you would see that they suggest the _minimum_ resolution to run it over is an x-height of 20 pixels, screens have seldom have one higher than 10 pixels. With those settings I got the following out of OPs comment:

     I've been really impressed with Tesseract - I used it last month to add invisible OCR text (1) to scanned PDFs I reference a lot. My scans
    are quite good, but I was still impressed with the accuracy.
    
    I also OCRed the TOC, playing with the page segmentation setting (2) in the terminal until I got output I could copy & paste to adda
    navigable table of contents.
    
    1: with the help of https://github.com/ocrmypdt/OCRmyPDF
    
    2: https://tesseract-ocr.github.io/tessdoc/Command-Line-Usage.h..., “ Using different Page Segmentation Modes”

I OCRed an unprocessed screenshot from the chapter's table of contents (1), which gave me (2). The collated table of contents (3) was error free, but as your example shows, this OCR isn't good enough to not need checking and proof-reading.

1: https://nexus.armylane.com/files/vogue-sewing-11-toc-screens...

2: https://nexus.armylane.com/files/tesseract-ocr-output.png

3: https://nexus.armylane.com/files/vogue-toc.txt

Tesseract was originally created by HP, open-sourced, and later developed by Google. It's based on techniques from the 1980s and is pretty underwhelming. But at least it's free!

When I read somebody praising Tesseract I always wonder how their experience could be so different from mine. It's so much worse than what you can do with a modern phone.

What is currently state-of-the-art when it comes to detetcting handwriting from photos?

Tracing strokes is nice but I would be more interested in converting my handtaken notes to markdown.

I don’t know if they are the state-of-the-art, but handwriting recognition in iOS and ChatGPT do wonders for me — even with an ugly handwriting. Though these are more like 90% to 95% accurate, you should review the output before trusting it.

I'm looking for something like that, but free/open source and offline. Suggestions welcome!

Its pretty remarkable. I've used my phone to take pictures of stickers with model information that I couldn't otherwise reach and was able to copy the text from it. Really wild stuff.

Very interesting experiment. I've been working on a handwriting application [0] for the past couple of years and incorporating the ability to take a picture to convert it into digital ink would be really nice.

[0] https://scrivanolabs.github.io

Can it read the scribble of my doctor? If so this is groundbreaking in the medical data entry space.

The number of deaths attributable to misread treatment orders in hospitals is staggering.

I'd be very careful about sticking another layer of interpretation between doctor and treating nurse.

It's unfortunate med school doesn't teach block lettering like they used to teach to draftsmen/women.

Almost no medical school or hospital in the United States use paper or handwriting anymore. All orders are electronic.

It says a lot that the illegible writing killed people yet appears to be affected by doctors in the same manner as a bimbo uses a vocal fry affectation.

Glad to see my scripts from the doctor these days are typed not written.

"Vocal fry" is a term some people made up to sound fancy. Actual linguists call it what it is: creaky voice. And they don't denigrate it. Men do it just as much as women, and there is (of course) no correlation with intelligence.

It is a very lazy stereotype, and (of course) completely wrong, to associate "vocal fry" with "bimbos".

Entering this sort of data correctly should be on the doctor. McDonald’s digital order style giant pancake buttons with huge touch targets with wide margins and large type on a huge touchscreen should solve the boomer objections.

Alternately, make it an app for the huge iPad Pro to solve the same problems. Make it as hard to fuck up as a fast food order. Disambiguation of input commands is a solved problem.

There isn’t really the will. Stupid windows apps with standard windows UI dropdowns that confuse and frustrate people not well versed with computer UI, running on standard low contrast small type displays with standard keyboard and mouse input, running on laggy RDP thin clients to underprovisioned workstation VMs in a data center is sadly the industry norm, and it still kills people.

It was a scary moment when Apple Notes corrected my writing in my own hand writing.

I still look forward to have programming environments on tablets that are able to use pen input, instead of forcing us to carry a bluetooth keyboard.

Apparently not something that anyone cares as business opportunity, because most likely most folks wouldn't pay for it, sadly.

I'm definitely faster at typing than writing though — especially so when it comes to something like code that often requires in-place editing, shuffling statements around etc.

I do like to work with paper & pen but moreso for ideation, diagramming, or todo lists rather than more "structured" inputs.

Naturally the idea is to explore new concepts a mix of diagrams and coding,

Basically what Brett Victor in recent times, and others have been trying to do for decades,

"Grail on Rand Tablet from 1968 demo"

https://www.youtube.com/watch?v=2Cq8S3jzJiQ

Or "Programmable Ink" at Strange Loop 2022,

https://www.youtube.com/watch?v=ifYuvgXZ108

There used to exist a graphical based Lisp for iPad, Lisping, unfortunely the app is long gone as it wasn't kept up to date with the OS.

Here is on old thread discussing it, https://news.ycombinator.com/item?id=3802131

The new iPad Calculator, is a more recent example of such ideas.

Man that would be horrible UX, I wouldn't pay for that. I'd only use it if someone paid ME to do it

Different strokes for different folks.

Can you touch draw 120 WPM?

No, nor do I care to type that fast.

Have the secretary typewriter contests from the 1950's come back into fashion?

They have in the form of monkeytype wpm scores. I think these are generally based on an a-z lower case test, so I doubt it's a realistic indicator of someone's actual typing speed.

I can't imagine people code at an equivalent to 120wpm.

A model that could turn "offline" handwriting (the ink on the page) into "online" (order and timing of the strokes) I think could be really useful for a historical HTR pipeline... but ultimately, we need end to end.

Why is historical HTR so neglected in all multi-task model evaluation benchmarks? There are millions of un-indexed handwritten historical documents which could give us a so much better understanding of our recent past. For that matter, it could give models much better understanding of our recent past.

Can we replace / augment the keyboard with a white (or marked) paper to write on ... that is in view of a camera that has real time OCR

Feels like it will be a good addition to input devices

Can this be used to create deepfake forged signatures/handwriting?

That was my initial thought too. For some important authors, there are a few high quality scans of their manuscripts, so with a tool like this you could create fake manuscripts -- and in a few years, after they are dead, say that you find them, create provenance, and boom, unpublished novel by JK Rowl- any author.

No. It is not designed to be used as a generative model.

But other people have created handwriting generation models.

If it can generate accurate shapes from a photo of a signature, why would you need a generative model? Just draw the SVG wherever you need it.

Eric Hebborn (https://en.m.wikipedia.org/wiki/Eric_Hebborn) wrote "Italico per Italiani" (available in Italian only, alas https://www.amazon.co.uk/Italico-italiani-moderno-trattato-c...) as a textbook/manual to improve day-to-day, practical calligraphy. Maybe there is something similar in other languages too...

i still write notes daily and already this year have finished 4 notebooks...however, things i need to review are typed, i write to keep things in my mind as they are being discussed...example, if i encountered an error while reviewing a program execution i would write it down "encountered error during attempt to do x" but i would also type it in my notes in vastly more detail with screenshots and other points...handwriting to me is almost like tagging it in my mind so i just don't forget that it happened.

Its not the writing PART!

It is the non automation part!

The experiment you can do to verify my point:

1. Write code by hand in plain text editor.....

What you notice....programming syntax and knowledge becomes easier to retain and re-learn.

No joke I do this several times a week.

As a lay reader, it's fascinating to see how much oomph we're getting out of LLMs on non-language-related tasks by figuring out clever encodings/linearizations.

This is very interesting. I had this idea of imitating human handwriting in my bucket of todos for machine learning models, but never got to it. I guess we aren't far from it.

I'm wondering how well it would work with my crappy writing and the fact I write some letters revere from normal

seems overkill... for a bunch of reasons tbh. I guess the golden sledgehammer is just too tempting.

taking photos of notes is a weird compromise that nobody really wants... we need better tools not better post processing

I don't think this is a hypothetical use case, at least for me. I like writing on paper with a fountain pen. But would like a digital version of the notes that are searchable. Reasonable ocr exists for conversion to text, but this would may be give slightly more accurate results.

such an exciting research project! I can imagine the impact this could have on education, e.g. handwriting notes of teachers in digital copies; or even preserve old documents in their digital counterpart

Can this work on low-power devices like note-taking tablets?

You may be able to run it on an iPad, but probably not a Kindle.

But also, the point of this is to convert photos of text into pen strokes. A note-taking tablet doesn't need it because it already has the pen strokes you wrote.

Can I just get good OCR for handwritten text? The last model that claimed to be "the best" was atrocious, only worked on PDFs of papers. ChatGPT is pretty decent but I was hoping for an offline, tailored solution

Check out textify on appstore. I have found the accuracy to be great even for my terrible hand written text.

> We present a model to convert photos of handwriting into a digital format that reproduces component pen strokes, without the need for specialized equipment.

Call my a cynic but this feels like a free way for Google to pull more data for training.

Sure. You're wrong and a cynic. The model does not connect to Google servers (unless you decide to use their colab, which is optional), you can use it offline without contributing anything back in data or code.

Aside from Google having published open source OCR tools like Tesseract for 20 years, it's a thoughtless accusation in general. What exactly is the insinuation? "Training" is just thrown out as a bogeyman. I can't even come up with a fictional scenario in which Google does something nefarious with piles of handwritten documents they've somehow acquired.

Training doesn't have to be nefarious. Google ran GOOG-411 in large part to collect speech data, but folks using the service still benefited.

> I can't even come up with a fictional scenario in which Google does something nefarious with piles of handwritten documents they've somehow acquired.

I never said it would necessarily be nefarious, but it's the same behaviour of data collection from users of free services to benefit themselves financially. While not always being particularly careful with collected user data.

A slightly related topic is around Google's training on YouTube subtitles. They're able to do this because they host all the content, but they dont allow owners of that content to opt out of that. Again, a free resource that Google get to play with as they feel like.

The linked project is an academic paper published simultaneously with open source code and a pre-trained, locally runnable model.

There is nothing that collects data here, and this is not a Google product that users interact with. Hence, there is no direct financial gain from this work. Google Research has published over 10,000 papers, few of which directly impact the commercial side of Google services.

Your example of automated captioning for videos doesn't seem particularly objectionable. Does it translate, indirectly, to slightly more ad views? Probably, but it's a rounding error in their revenue. I am guessing that few content creators who publish on YouTube find this feature controversial: In addition to free hosting and revenue sharing, they don't have to bear the costs of writing captions while benefiting from accessibility and discoverability.

There are valid complaints against data collection by these tech companies for machine learning. Artists have a point when they condemn generative models trained on their work. And you might reasonably object to the collection of handwriting samples that Google used here, which were scraped from public Imgur posts (Facebook Research's Imgur5K data set).

But there's room for nuance in deciding what uses are fair and acceptable without the knee-jerk reaction of tech company + AI = evil financial motives.

There is a repo with Apache license. Does that need to connect to Google services to use?

No evil.

That’s exactly what I thought.

finally, I will be able to read my doctor's prescriptions.

Now I'm wondering if they can use a similar architecture to derender paintings. It would the quite something to have a stroke perfect recreation of the Mona Lisa drawn by a modified pen plotter.

Rings a bell: https://arxiv.org/abs/2108.03798

Tbh my handwriting is write only. Through decades of computer use it has deteriorated to the point were it is illegible even to myself.

Still. I will always be notetaking (and doodling) during meetings as it helps me order my thoughts.

Still. I will always be notetaking (and doodling) during meetings as it helps me not fall asleep and makes it look like I am paying attention.

Same here.

I learned 10 finger typing when I was 8 in school (42 years ago), and have been glued to computer since then ; if I write something, or even sign something, it is complete gibberish, even to myself. To me it looks like those last stages of dementia scribbling, but I know that's not it because it was like this since I was 12 or so; my teachers couldn't read my writing and it affected my grades.

I’m the same way, and between that and the fact that I can type 120+wpm, makes me angry every time someone expects me to write something. Making dirt smudges on dried tree pulp is as archaic and outmoded as carving marks into bones, and in my view has no place in modern society outside of art.

> in my view has no place in modern society outside of art.

For a lot of use cases I agree, but no place is to ... abrupt in my opinion. Computers for sure have a multiplier affect in many cases. But when you really want to think something trough it's advised to slow down. For those cases paper and pen is the perfect medium.

And I have a personal preference when making notes with pen and paper about hikes and climbs I'm working on/doing. But that's just me

> But when you really want to think something trough it's advised to slow down. For those cases paper and pen is the perfect medium.

Or chalk on blackboard/marker on whiteboard. A marvelous way to get thoughts flowing that just don't seem to otherwise.

I’m pretty sure most modern mathematics research takes place on pencil and paper or chalkboard. It’s too tedious to typeset everything in the research phase. It’s also way harder to draw little diagrams or new invented notations on the computer.

If mathematics was as young as programming, it wouldn't use any fancy symbols and make everything ASCII compatible. Though that wouldn't solve diagrams...

Same goes for my signature, I almost get anxious when I have to sign something on paper.

No worries, signatures are expected to illegible scribbles.

Funny how it almost becomes its own language, legible only to the subconscious

Try Kate Gladstone's "Handwriting Repair" course:

https://handwritingrepair.info/

the biggest thing which helped my handwriting was switching away from ball points to either felt tips or fountain pens.

Getting a Newton Messagepad also helped markedly, and since then I've been using tablets w/ Wacom EMR styluses where possible (NCR-3125 running PenPoint (donated to the Smithsonian by the guy who bought it) through a Samsung Galaxy Book 3 Pro 360) and applications such as Nebo or Write by Stylus Labs.

You can fix it if you want to.

By the time I graduated from school, my handwriting was pretty bad. I could read it, but nobody else really could. Then I got a Palm Pilot and it's primitive handwriting recognition forced me to slow down and make better letter forms. At the same time I devoted a little effort to improve my cursive as well.

It's stuck with me and my handwriting is still fairly easy to read. My block caps look like something off an architectural drawing thanks to Palm's Graffiti system beating me up nearly 30 years ago.

Same. Over 20 years (note taking especially) it is reduced to scribbles. The value of it is the time thinking while writing it mostly, though todo's are longer lived.

If your todo list is, or needs to be long lived, I think you might have more problems :D

I make notes on paper as the act of writing makes it stick about in my head longer than it would if I typed typing them out; though similar to you there's better penmanship from a spider that fell in ink.

Long lived may be tomorrow/a few days :-) And struck through. And written at any angle on the paper. Definitely not a codified method for me.

Use boox or remarkable! You get both advantages of having the writing experience and their automatic recognition turns your scribbles into text!

I like this 'my handwriting is write only'.

when i was 18, i looked at notes from my new coworker / manager and asked him how he learned to write that nice. He told me that he struggled writing cursive and just started writing letter by letter (print letter style?).

I changed my 'font' on that day and suddenly i was at least able to read what i wrote!

Interesting.

We grew up learning to write separate letters.

Cursive was taught only very briefly as a way of telling us tgis also exists but don't bother too much about it.

Nevertheless, cursive is where the speed is. University times I was writing at the speed of the professor chatting and still having it halfway readable even for others - even though the looks of it were tending to the Arabic (I'm Latin). Good times, now I can barely scratch a shopping list before I break my wrist.

Decades of typing, and now my handwriting is more of a cryptic art form than anything readable

Even my mother who is 68 has severely deteriorated handwriting. Nobody writes by hand in a Western country since the smartphone.

It's interesting that for decades, technologists were stubbornly predicting that the paperless office was just around the corner:

https://en.wikipedia.org/wiki/Paperless_office

These predictions were famously overoptimistic. Paradoxically, paper consumption went up for a long time instead, because printers became cheaper or better, and wysiwyg word processors made it far easier than type writers to produce complex documents, with pictures and diagrams and tables.

But approximately since the smartphone era, printer usage indeed seems in sharp decline. The paperless office, at last, became reality. Just a few decades later than expected.

If I could get a digital music stand that worked well and wasn't expensive I'd get rid of a lot of my print. (I need one for the piano, one for the keyboard, one for each kid into music, so cost is very important!)

I feel like even if I try to write neatly, it’s like my hand has forgotten how

So you can't explain something to people on a whiteboard? I've been typing since I was a child but I'm still perfectly able to write legibly. If I thought my handwriting was that bad I would try to fix it as not being able to whiteboard would be terrible.

Many of us don't do physical whiteboards anymore. I diagram a lot, but all of it is online. Apart from own satisfaction, there's no reason for me to get better handwriting. (And there lots of things more satisfying than that) Addressing a letter once a year without a printed label, I can use block letters.

Same here. And not only that, years of keyboard and mouse usage have done some damage to my wrists and nerves and it's actually painful for me to write.

I believe that pain comes from misplaced expectations. We remember the times when we were fluent writing, or see it in others, and expect our hand to follow the same pattern with the same speed. So we push too hard. We should realize instead our hands must re-learn to write at this time, drawing the letters so slow like the small children do.

I spent a few years doing this without google.

I suggest it, and I suggest it without google.

Reply on news.ycombinator.com