How to digitize your lab notebooks | Hacker News Reader

How to digitize your lab notebooks

nature.com

118 pointssohkamyung83 comments

Here's what I did, which works much better than phone apps or scanners.

Mount a digital camera on a tripod and bring a couple lamps over for strong uniform lighting. Turn the page, take a picture, turn the page, take a picture. You'll go through a 100 page notebook in 10 minutes (not an exaggeration). There are better quality ways to digitize documents, but I don't know of anything cheaper and faster.

Even some of the best quality scanners mimic this technique

I don't have a link handy, but this is what Google did when digitizing books for Google Books back in the day. They had a hardware setup that I believe had foot pedals to snap the picture, but human hands are (were?) still the best page turning technology they could muster. There was also software to make the image of the page appear flat, rather than curved as most books appear when laying open.

yeah! I just remembered there were some news articles about hands appearing in some of the scans

> There was also software to make the image of the page appear flat, rather than curved as most books appear when laying open

Yes, they projected a grid (using red light, I believe) on the page so they could compute the curvature to re-project the image as "flattened".

Interesting. I had been led to believe that Google used a linear book scanner towards the end of the big Google Books library scanning project. It looks like this may have been an internal prototype that wasn't necessarily used for large-scale scanning opert

https://www.youtube.com/watch?v=4JuoOaL11bw&feature=emb_titl...

https://hackaday.com/2012/11/16/google-books-team-open-sourc...

https://code.google.com/archive/p/linear-book-scanner/

https://linearbookscanner.org/

There's a guy that build a high end DYI book scanner using this same idea. He sells them prebuilt every once in awhile as well.

http://diybookscanner.org/archivist/

On the DIY Book Scanner forum, I saw the TIFLIC scanner, which I plan to build. A build post: https://evilgeniuschronicles.org/2018/09/25/my-build-of-the-... A demo from the creator's about 1 minute in: https://vimeo.com/213003462

It feels simpler to build and it takes up less room.

This seems similar to the rocketbook, but more permanent and less reusable.

Fujitsu ScanSnap SV600 is a hardware setup for such scanning.

This is an interesting article but I'm as much or more interested in the bigger picture of what to use going forward. I haven't found the killer app for lab notebooks. Any ideas form the community?

Requirements:

  * simple formatting (ala markdown)
  * ability to share with colleagues
  * able to simply paste in images of graphs etc..
  * versioning, ideally with ability to timestamp
  * tables
  * code snippets (with formatting)
  * links, tags, backlinks

versioning is useful to be able to see how knowledge on a subject was updated over time. There is a second level, important to some users though not necessarily me, where physical notebooks use for patents/IP to establish the date of an innovation.

OneNote works on most of the above, but:

  * its search is terrible
  * its formatting of code is terrible
  * easy for people to pickup and use

I've thought about a note-taking app like Obsidian, together with git. But it isn't easy to share notes. its handling of inserted/pasted images is a little jank.

Immutability of information in lab notebooks is also important for auditability. Pen in a bound paper notebook works well for this. Missing pages or erasures are a huge red flag for inspectors and rightly so.

Wouldn’t git with signed commits work just as well? You’d have to find a good app that saved stuff as text (not Word, but probably not LaTeX either since that takes time to get the formatting right), but once you figured that out, you’d have a full history of all changes made.

Yup - that's probably a good equivalent.

Signed commits still let you falsify your own work after the fact. You need an external trusted service, a digital notary essentially, that would provide a signed record that you submitted a commit on a certain date with certain set of ancestor commits. Each time you added an entry, you would submit the hashes of all the entries in the book to the digital notary and get a signed record that you submitted those commits on that date. Inability to produce this record would be similar to having erasures or pages torn out in a lab book.

a public blockchain would do for the digital notary? or hash posted on twitter?

You can do this in a pretty cheap and low-tech way by publishing the hash of your current state somewhere that will keep a permanent record. E.g. the Bitcoin blockchain or a classified ad in a local newspaper.

This. True lab notebooks are not just personal notes, they are a legal record. Not every page of every lab notebook will matter, but it absolutely happens and is one of the many real use cases for keeping a lab notebook. If you invent something or discover something of significance, you will want this. Paper notebooks have pages stitched together and ink that can be run through chemical analysis to see if all the text on the page was written in a single session by the same pen. Forensic examiners have about 100 years of experience assessing paper documents. Digital systems can provide legal grade archival support but a sweet front end editor with copy paste and markdown support does not imply anything about traceability of the contents.

A VCS like Fossil at least makes the parent commits un-fungible, so you would only need to sign the latest commit, and not every commit every single time. Git is amazing in it's flexibility, but it's not necessarily always the right VCS solution.

Otherwise I agree with your answer about Git.

Yep! This is already done and for exactly this purpose; I believe the common term is "trusted timestamping" [1].

[1] https://en.wikipedia.org/wiki/Trusted_timestamping

That seems difficult unless it is completely automatic. Also, you will have to convince auditors that your implementation works and that your users are using it correctly and that everything has been committed correctly and in the correct order. So while it might be obvious to you that this works "just as well" it is far from obvious to others.

You wouldn't need to sign the commits, you just need to give the latest hash to a trusted (by the authorities) third party, and that will validate the existence of all history up to that point at that time, so ideally at the end of each day, you give your SHA-1 HEAD to the authority.

This is basically what the block chain does, except that instead of having somebody designated as special and trusted, they have the signers perform proof of work.

I remember that i read about a lawsuite were someone tweeted the hash of a document and proofed that way that this documenta existed at that point. I believe it was about a patent infringment and they even brought a twitter employee in as a witness to testify aboht the imutability of tweeta and the timestamp.

Cant find the article (hash is not a good search phrase when it comes to Findung an article about twitter).

This is the case in some fields, particularly life sciences where it's very difficult to reproduce results, but in others it's not important at all. I have colleagues who have to have their labbook pages signed by the group leader each day, and others that can even take their labbooks with them when they leave their jobs because no one minds/cares.

Indeed, and different fields operate under different regulatory / legal environments. When my parents were working as research scientists, they followed a rigid set of guidelines that were considered to be the gold standard at the time for use of notebooks in patent disputes.

Labs in regulated environments such as medical or pharma research have their own rules today, and an industry of vendors who sell enterprise level software for managing data in a compliant fashion.

I was working at a company on the day that US patent law was reconciled with the rest of the world, and the rules for priority changed. That was the last day we were required to keep a lab notebook.

Today, I mainly keep records so I can figure out what I did later on, and the best single tool that works for me is Jupyter.

Interesting, what about the fact people can just not write something in it?

I believe eLabFTW (https://github.com/elabftw/elabftw) would fit ALL your requirements.

Over the past few years I've developed a WordPress plugin that provides most (all, I think) of what you want [1]. I also a while back did a comparison of some other softwares' pros and cons [2].

At my work we've used my system for about 2 years now and there is much more engagement with the system than the one it replaced (a custom Lotus Notes application... go figure). People like that they can open the logbook from the lab, write some stuff, save it and edit/publish it later. We also heavily encourage to document anything and everything and use the categories and tags as much as possible to make searching easier.

[1] https://github.com/Academic-Labbook/alp

[2] https://alp.attackllama.com/2019/02/05/labbook-software-shoo...

My university has supplied labs with labarchives (and has guaranteed if it ever migrates away from labarchives it will provide an equivalent solution and handle migration). It ticks all of those boxes. My only gripes are minor. Things like no embedding of vector graphics and no good embedded spreadsheet solution. They have some MS office browser based solution that handles version control. That is the official best spreadsheet solution right now.

I generated an html spreadsheet for run logs because I wanted an embedded setup. I ran into the limitation of 64K characters per entry. It’s handled by trimming html off the end then autofixing broken stuff. It works so well that it took me a while to figure out why the number of rows was decreasing as more rows were filled out.

It’s all very tidy once enough people are on board. Whole notebook (and subsections) can be exported to html or PDF. There is even a way to have some robotic file uploading a la directory monitoring.

Have you ever considered Kyso (https://kyso.io) (disclaimer - I'm a co-founder)? I could be way off, and it would be a corner case for us right now, but it seems we'd fit all your requirements:

* We sit on top of git, which remains the single source of truth for your code - versioning, reproducibility, etc.

* Python & R notebooks or simple markdown (also on the web app itself).

* Show or hide code.

* Simple links for sharing reports.

* Tagging.

* Comprehensive search - by tag, keyword and content (soon).

Would be really interesting to hear your thoughts on the fit.

Markdown Enhanced for VSCode[1] + Git for versioning. \

[1]: https://shd101wyy.github.io/markdown-preview-enhanced/

Maybe I'm missing something, but doesn't org-mode (on top of git or another VCS) satisfy all these requirements?

If you don't mind relying on a third-party, Notion [1] has pretty much all of your requirements.

Formatting is a pleasure (including formulas and LaTeX), tables are decent, you can easily share notes, etc. Code formatting can improve a lot (only basic language semantics/operators/strings); search is decent. Above all: it's incredibly simple to use, and very customizable.

My three big missing features: 1) full API, so you can develop extensions that provide the missing functionality, 2) end-to-end encryption, 3) 2FA.

For now I'm using only for study lab notes, so encryption is not a big deal. But I wouldn't feel comfortable adding anything confidential without a more decent security model.

[1] https://www.notion.so/

but it's a proprietary format and not stored locally, unlike obsidian

Re: not storing locally, yes, hence my first comment on relying on a 3rd party. It depends on what is your intended use, given Notion doesn't have e2e encryption yet.

Re: proprietary, it exports natively in Markdown (including a zip with all images), or pdf, or html. It's not perfect (there's a bug when exporting some LaTeX formulas in .md), but for the most part it works well.

When I was in the sciences, I used a LaTeX document as my labbook

* Ability to share with colleagues - Easy to send pdfs or the raw LaTeX to others.

* Able to simply past in images - I had a folder with dated images, and a macro to insert a graph at the current location (better than floats for labbooks).

* Versioning - Each section had a date in it. Sequential versions were stored in git

* Tables - `\begin{tabular}`

* Code snippets - `\begin{lstlisting}`

* Links, tags, backlinks - `\ref{section_name}`

It could do pretty much anything. Though, you may have noticed that I omitted addressing your first bulletpoint of "simple formatting", as I am not able to keep a straight face while calling LaTeX "simple".

> It could do pretty much anything. Though, you may have noticed that I omitted addressing your first bulletpoint of "simple formatting", as I am not able to keep a straight face while calling LaTeX "simple".

One possible way of harnessing the power of LaTeX while keeping a simple formatting would be to use use `pandoc` and pandoc-flavored markdown. That keeps the formatting simple, while also making it easy to include references, equations, etc. and easy output to LaTeX/PDF/etc.

Why are so many people interested in markdown for notes? I've never used it before but HN seems really interested in note taking and in using markdown for note formatting. What does it offer that OneNote or Word doesnt?

I think the difference is probably that markdown is a pretty transparent and versatile way to format things, while still being simpler than many other markup/formatting languages.

With Word, you might insert an image or try to add bullets to something and end up with your whole document looking completely different, whereas, with markdown, you know that if you put in a ">" for a quote, all that's going to change is that one spot.

I think it’s formatting as code vs WYSIWYG.

Good point. Looking back at my comment, that's basically what I meant. I just used a lot of extra words to say it, I guess...

I'd like a speech interface, as I often have my hands full with actual lab stuff.

I think this is a problem not just for the lab community but for digital tools in general. There really isn't a good single app for doing many different styles of thinking and representation (text, drawings, formulas, tables, photos), organizing those things, and sharing them. Many apps get part of that problem right, but not all of them.

Wordpress? Don't know if Blogspot or other blogging platforms offer it, but I'd say Wordpress covers most/all of them.

Plus, easy to import/export data.

You can do all of this in rmarkdown and some grep. git integrates well with rstudio too. plus an rmarkdown file or notebook is a plaintext file so it's future proof. It's what I use and I love it.

What specific version of one note are you using? The search in OneNote included with office is noticeably better than the OneNote for Windows 10 that is preinstalled. There is an addin, NoteHighlight2016 that adds code syntax highlighting. OCR for text within images is another helpful feature that is missing from many alternatives.

Can anyone recommend a Free and Open Source tool good for this sort of digitizing work? Presumably transforming photos of documents into PDF files with OCR annotations, though I don't see that it has to be PDF. Decent handling of formulae would be a plus.

Also, the website's cookie-consent dark-pattern leaves a bad impression.

If you are fine with the command line, then nothing beats some combination of Ghostscript, ImageMagick, and PDFtk. OCR is much more complicated but I have used Tesseract when OCR was needed. The results were mixed at best, sometimes being excellent and sometimes being terrible. It largely seems to be a function of the quality of the scan (how flat the text is, how horizontal the lines are, ...), though even seemingly minor deviations sometimes result in unreadable OCR text.

https://zettel.hackstream.dev

That seems to be a note-taking tool, not a tool for digitising written notes.

This guide [0] is intended for digitizing books, but it can lead you to some places. If you can get your scans to a PDF (e.g. with a scanner or a smartphone app), you can always try OCRmyPDF [1] to perform the OCR for you.[2] has some tips, but try to look for an older version in Wayback Machine, as I think it has been condenses for the worse.

[0] https://www.natecraun.net/articles/linux-guide-to-book-scann...

[1] https://ocrmypdf.readthedocs.io/en/latest/

[2] https://colinpurrington.com/tips/lab-notebooks/

I've written about the tools I use here: http://www.davisr.me/projects/books/

Great write-up, thanks.

I'm finding that ctrl-f isn't working though. When I search for Since Apple it doesn't find the match on page 21. I've tried this on both Firefox and Edge (Chromium version).

In an ideal world, it would also 'clean up' the text, replacing the image with 'live' text in the appropriate typeface, but that's more a wishlist than a real problem.

I found this gent's articles quite helpful:

http://blog.konradvoelkel.de/2010/01/linux-ocr-and-pdf-probl...

http://www.konradvoelkel.com/2013/03/scan-to-pdfa/

I don't know how researchers feel about spending over a grand on it, but an iPad pro lets you write in free form, draw diagrams, etc. You can take pictures with it too. You can even connect it to Google Jamboard and then all your stuff is online and shareable.

I'd definitely have preferred to have one while I was studying.

I tried for a while with a convertible laptop and a stylus that goes with it. In the end it was too bulky and when i spill i might kill it. I am back to the paper notebook for bench work. All data and documentation of results goes into Confluence. This hybrid served me well lately.

My daughter started at university this fall and she needed a computer. She could have picked anything (within reason) but what she wanted was a big iPad Pro with the keyboard. She's in a life sciences program and so far it's worked perfectly for her.

Which note taking app do people use on the iPad?

I like the idea of writing notes with the Apple Pencil and converting some of them to text for readability. The converted text also allows me to fit more on a page.

Anything allow embedded Latex?

Notability is pretty good. It's not too fancy, but it's very fast and Ginger Labs has continually added new features.

My daughter uses GoodNotes. You can convert selections to text, but I don't think there's an easy way to bulk convert. Behind the scenes it's doing it because search works on all handwritten notes.

Some of the apps allow for infinite scrollable pages so maybe that would decrease your need to convert everything to text?

What problem does embedded Latex solve?

I believe the bundled Pages app from Apple can handle Latex.

Latex is a nicer way to write equations.

You can use a sigma character, for example, and move characters around to create equations, but it’s cumbersome.

Plus, it’s a nice little language for formulas. Someone else might give you the Latex and you could copy and paste that into your notes.

I can see that if you are typing your notes, but when using something like an Apple Pencil, you can just draw a sigma or whatever symbol you want.

yes, and in the past that’s what i have done:

https://github.com/melling/MathAndScienceNotes/blob/master/s...

My point is that when you want to make your notes look neater, replacing your handwriting with Latex equations would look much nicer.

I can confirm that Pages works with LaTeX, but Pages is not really a good alternative to Notability or GoodNotes.

thinking about an e-ink tablet to avoid long-term eye damage like supernote, onyx, remarkable

What kind of eye damage are you talking about?

blue light and front light strain

I have tried iPad Pro a few time and I find myself just cannot use it. I still prefer my trusty notebook + fountain pen.

Personally, I also maintain a Big Text File as my digital note. So I put the index to my physical notebook in it. That way, I could do digital search in my digital note, and refer to my physical note if required.

Are the file formats either open or exportable to something that is open? If they're not, you're in the exact same place as the author.

It depends on the app you use to take notes. The one I use (notability) can export files as pdf.

An ipad would never survive a lab environment from falls alone, and thats not even mentioning liquids. they only last a few years under the best conditions before the non-removable batteries crap out anyway. Plus total lack of repair options, no keyboard, and being overpriced makes them a hard sell. We wouldn't buy an inferior power supply or particle detector for double what its competitors cost, so i can't imagine blowing that money on a lab ipad. Many people i know use the used thinkpad x220 around the lab for computing, but there is always a proper lab notebook. If experiments from 20 years ago had stored their only lab notebook copy in a file, it's a practical gaurentee that it would be lost by now. Things are even more complicated and brittle in 2020. God help you if you rely on cloud storage for essential information unless you're prepared to watch like a hawk for the company's innevitable demise and 30 day deletion notice. There is still no realistic substitute for a shelf or cabinet with old lab books on it. Keeping physical lab books can also be a requirement of the grant.

Lab books typically don't do well liquids either. At least with an iPad your data should be safe if it gets destroyed.

Also, iPad batteries and screens are replaceable by Apple and you can definitely attach keyboards to them. Even if they did only last a few years, that's only a few dollars per day. Your lab probably spends more than that on coffee and tea for you.

Must be nice. I've never worked in a lab where coffee and tea were provided.

My lab books from the 1980s are still in fine form, because lab books are precious so people look after them. (Also, I was scared of losing something, so I photocopied them a month at a time, and stored the copies in a different location.)

Since I put a table of contents at the start of each book, with a small set of keywords listed by every title, I can find things pretty well.

A lot of people on this thread have stressed the importance of immutability, and that is really significant. When I found an error, I would make a new entry (on another page, maybe in another book) that corrected the error, and added information about this to a changelog.

Oh, and I entered the table-of-contents line into a plain-text computer file for each entry, so I could search for things by phrase or keyword (grep is your friend).

With a notebook with high-quality paper and a pen with good ink (I used India ink a lot, which is waterproof), the notes will easily last 100 years, maybe 1000 years. Do I think Apple's ipad format will last that long, or that Apple itself will? Not so much.

An issue in academic labs is that the funding agencies forbid spending grant money on computers. So the academic workers pay for their own computers out of pocket, and are quite unhappy about it.

They buy the cheapest computer that will do the job. Now, to make full use of the university computer network, their computer has to be "set up" by the department IT person, resulting in a significant degradation of performance, possibly installation of software that you don't want, and the inability to do your own maintenance. Often the computer is too old to support the latest OS, so it can't be "set up" anyway.

So they typically connect to the campus guest wi-fi network and use things like Dropbox to save data.

Now you can buy a computer with grant money, if it's purchased with a piece of scientific equipment. So the spectrometer in daily use, well, it was purchased 20 years ago, it only runs Windows NT, and can't be connected to the network for fear of breaking it.

The data have to be accessible on a different computer at home. They can't take the same iPad home because it might be radioactive.

Nobody's going to buy an iPad for the lab.

I've known plenty of labs buying computers for students and staff with grant money. I don't think there is a ban, at least in my field (biology). It's a tool like any other, and they aren't full of bloatware unless you mean access to the university intranet if applicable.

It may depend on the specific funding agency. I know that my spouse has to provide her own computer.

I missed the edit window, and of course I was remiss in not looking this up. It was something that I've heard from multiple people, but never verified. Here's one reference:

https://resources.finance.duke.edu/research/documents/eTutor...

Though it doesn't help with someone trying to keep their notes in a bound notebook, I'm a huge fan of scanning notes and papers and here's my setup:

  1.  Scan with a Fujitsu fi-6130.  This scanner is out of print and I believe the successors are the 7160 and 7180.  When I purchased mine, it was somewhere between $900-1000 and I've used it actively for about a decade with zero issues.

  2.  Use xsane for scanning

  3.  (Optional) Use scantailor-advanced (https://github.com/4lex4/scantailor-advanced) for any cleanup

  4.  (Optional) Use tesseract for any ocr (does not work on handwritten notes, but works great for scans of papers, bills, etc.)

Generally speaking, I take hand notes on research topics and the collect them till I need to travel or get too many. Then, I scan each small collection into its own file. To make this easier to back walk, all notes are dated, numbered, and given a general topic/title as I write them. Once they are scanned, they go into their own pdf file with a long file name that contains the date, title, and keywords that I want to use to find it. If there's a long thread of thought that needs to cross reference multiple files, I've added that information directly to the hand written note with a comment like (these notes use the following results from this date).

While not perfect, I've thousands of pages of notes and I've always been able to find what I've been looking for with little effort. I'm not sure anyone else could look at what I've done and search it as effectively because I kind of know how I think about problems and label things, but it works for me.

As a couple of other small tidbits:

  - It takes too long to individually fix each scan using scantailor.  A good scanner with a straight scan is *much* better than fixing things by hand.  I will do it if I have to, but I'm disappointed in myself when I do.

  - Slow scanners will kill this methodology and make things too burdensome.  Don't use a flatbed scanner.  Get a fast scanner with an autofeeder.  They are expensive, so expect about $1000.  That said, I've scanned tens of thousands of documents on mine and it's never failed, so a good scanner is worth the money in my opinion.

  - I have thought about using a tablet like remarkable to take notes.  My issue is that since I already own the scanner and paper notes are so incredibly robust especially when I'm on travel, I've been unable to justify the money.  Though, annotating pdfs may be nice and a scanner doesn't help with that unless we want to print, markup, and then rescan, which is a little wasteful.

Hi,

You'd probably like my site. I discuss both book scanning (using a Fujitsu fi-4120c with SANE)[1] and reMarkable hacking.[2][3] I've also written a popular GUI client for transmitting reMarkable documents without the manufacturer's cloud (sold under GPLv3+) that many people find useful in their workflows.[4] I used to take notes on paper and scan them, but after getting a reMarkable, it became just so much more convenient. I now use it for 100% of my reading and writing.

[1]: http://www.davisr.me/projects/books/

[2]: http://www.davisr.me/projects/remarkable-microsd/

[3]: http://www.davisr.me/projects/parabola-rm/

[4]: http://www.davisr.me/projects/rcu/

Funny enough, I have been watching your stuff. Very good work and its made moving to a remarkable more tempting.

The new ios14 / ipados14 has OCR on notes, works relatively well on my side. I just need to write patiently enough instead of my normal scribble.

Curious if there has been more adoption of these in the lab environment as well: https://youtu.be/-djt0IA4s0w

One super interesting startup in this sector: https://www.labtwin.com/ they completely automated lab reporting and made it voice-first

Reply on news.ycombinator.com