Keeping a Lab Notebook [pdf] (2013) | Hacker News Reader

Keeping a Lab Notebook [pdf] (2013)

training.nih.gov

118 pointsTomte47 comments

I have kept lab notebooks for decades now, some thoughts;

Writing them, on paper, has been very helpful. The act of writing, the pace of it, the composition of what I'm going to write, combine to help make the information more durable in my head some how. I've got a ReMarkable 2 notebook which gives me infinite pages which is kind of neat, but I still like paper notebooks.

They have saved me a couple of times and made me more productive. Since I often have a number of projects in parallel so that I can pick up one when the current one hits a stall (like waiting for parts, or needing different equipment), and when I come back to a project I've organized my notebook so that I can re-create my mental map from where I left off.

And in one instance it torpedoed a vendor who tried to patent an idea that we had reached out to them for help in developing.

I finally made the leap and switched to an iPad+pencil2. I’ve tried

Writing in it is not as automatic as my prior tool (cheap notebook and fountain pen, or whatever pen is available). Also portrait/landscape doesn’t really work. But over time I got used to the experience and this part is adequate.

There’s a property of physical book where you can flip back to some point in the past. I find this doesn’t work as well with ring bound books anyway.

It doesn’t have the whole “sign the page” property but I haven’t been subject to that in years so not a problem for me.

But it has several benefits:

1. It automatically converts and indexes my handwriting so I can search (!) for a note or mentions of a thing.

2. I can carry all my notes and they are with me all the time. I can even look at them on my Mac though they aren’t “live”.

3. I can connect to zoom a second time from the iPad and share a picture, or draw something in my notebook that everyone can see, and then it’s saved.

It was worth the switch. I considered a ReMarkable and a Samsung tablet but am glad I sprang for the iPad. I have tried the others (friends have them) but this seems the best for me.

Note: don’t use the iPad for much else. Sometimes I read rss in bed before sleeping if the iPad is in the bedroom for some reason.

I've done this as well. Because I was trying to get to that point ever since I got my first iPad I've got a number of different applications for "taking notes". The notepad integration with the pencil however is top notch. I had one that integrated with Evernote and came with a stylus (before the Apple pen existed) and it worked well too, but Evernote lost their way.

For me, the ReMarkable2 is actually a bit better for this, only because the drawing is more seamless and the battery life means I can take it camping without worrying about charging it every day.

In my paper notebook I use 'manual' doubly linked lists to connect places where I leave and return to a project/note. So when I start a new thing, it goes in the front contents with its starting page. When I come back to it, I write at the end of where I left of "to page->xx" and on the page where I've started up again I write "from page->yy". So that at any time I can find a project in the contents, go to the start, and then walk through all of the pages that have some thoughts or information about that project quickly.

The other thing that paper notebooks do well but electronic ones do not yet is flags. I have 'post-it' 5 color flags usually glued/taped to the inside cover of the notebook. So I can add flags to pages with specific things like 'green flag' means business idea, 'yellow flag' means waiting on something before I can continue, 'red flag' means this has an action item I need to do/finish to make progress, and typically 'blue' flags are used to indicate the first 'unused' page in the notebook so I can easily open to it. The 'Makers Notesbooks' from Make Inc have a ribbon page marker that works for this as well.

My favorite part is being able to doodle and erase it over and over.

Any time I take paper notes, the margins are filled with doodles. But on the iPad I can doodle and just erase and no one is any wider!

Im thinking about making the same switch. What app are you using on the ipad that makes your notes searchable?

Ordinary apple notes.

For a good historical example, see Linus Pauling’s notebooks (with page numbers and dates): http://scarc.library.oregonstate.edu/coll/pauling/rnb/

The blog by biologist Dr. Colin Purrington also has advice for maintaining a notebook (discussing why, backups, and what to note down): https://colinpurrington.com/tips/lab-notebooks/

I came here to add Purrington's blog. But I would also add that you should look at older versions in the Wayback Machine, it used to have a little more depth and images depicting things (e.g., a paper chromatography test of different inks).

Having spent 3 years with lawyers in Patent Litigation:

This stuff is old-school, but the lawyers all know & understand it. Courts understand it. If you do something in a way that someone else has already done and testified in court about it, you win. This is not the place to be innovative.

This doesn't apply anymore with first to file, unless you preempt the filer by publishing it publicly.

You still encounter old patents in litigation, though, before first-to-file was introduced.

It also has other uses beyond Interference proceedings.

This is where electronic notebooks really shined. No longer in the lab, but many experiments are repeats with slight tweaks to the procedure. Rather that rewriting everything out we’d often just write “same procedure as XXX-YYY with the following changes”.

It always felt much cleaner to just copy-paste an experiment and directly edit the changes.

Plus being able to type versus write meant I put way more detail down and could easily attach data.

It should be noted too that the US patent law changed from first-to-invent to first-to-file in 2013 (when this document was written). The importance of lab notebooks in defending a patent becomes much less important after that. I know defending a invention was a HUGE part of keeping a good notebook.

https://en.m.wikipedia.org/wiki/First_to_file_and_first_to_i...

Does anyone have experience using something like Obsidian as a sort of lab manual for software engineering? How do you structure your notes? What gets written down? I use Obsidian and the note of the day plugin which uses a template that has a todo list and a Recap section in which I intended to summarize the work I did that day and any comments worth writing down. I used the todo list part daily but I've struggled to consistently write the recap. I think it's too vague; I'm looking for a little more structure that makes recaping the day easier and manageable. When I try to write a recap currently, it can feel overwhelming as I try to think what I should and shouldn't write, and how I should or shouldn't write it.

I've been using TiddlyWiki for this for a few years. It started off as being my lab notebook and has expanded to be my everything-under-the-sun notebook.

It began simply. It started because I had a bunch of new services/apps being thrown at me on a new project on a bunch of new servers. I recorded the information and linked it all together and expanded it to include meeting notes (on the level of minutes), transcriptions from my physical notebook, research notes ("here are examples of the issue, what we expect, what we think it should be, all the stuff I've tried, etc."), architecture notes.

TiddlyWiki has a journaling feature that I use for the recap-level you talk about. I usually head the journal entry "Minutes of the Day." There will be a bullet point for each project/stream and then sub-bullets for each meeting (with top level decisions/takeaways under that) or thing I did. It's a bit subjective, but I keep these entries high level and will hyperlink to more detailed research notes.

I do, but not in any deliberately very structured way. I use daily notes as a high-level "this is what i worked on" journal, with links to the details of those items. For example:

  # Work
  
  Added feature [Foo](https://pivotal/...) to [[Bar Project]]
  - Learned about sprockets: <https://wikipedia/Sprocket>
  
  Talked to Jane about the state of [[Qux Project]]

As I'm writing it, it gives me quick access to the resources I'm using today. With backlinks, if I'm looking at the "Bar Project" page, I can see when I was working on it and what resources I was using on those days.

My process optimizes for low friction. I want to take notes, not necessarily spend a lot of time rigorously organizing notes. It also gives me a sort of the timeline view that Zettelkasten fans like, but without the rigorous tree structure that I don't personally seem to benefit from.

I've had relative success with keeping it open + on all desktops. I don't fullscreen anything, so it's always visible - since I started doing that, I'm MUCH more consistent about updating it.

Prior to that, I had this small script that'd open textedit to a daily-note every time I made a terminal session, which served much the same purpose: https://news.ycombinator.com/item?id=22279802 . Obsidian makes it a lot easier to jump between days though :)

(as a semi-aside: I very much like Obsidian. It's one of the only "just let me use my pile of normal markdown/commonmark files like I want" apps I've found, after trying dozens. it's very very very unintrusive / doesn't impose any structure. highly recommended!)

Slightly tangential, but does anyone have a story for how to share Obsidian notebooks in a team context? That's the one major thing I've found missing in these off-line first notebook apps.

Obsidian uses markdown files stored in a folder, so any fileshare should work fine. There's no account management or permissions needed, either you have the .md files or you don't. I use Dropbox to carry notes between my PC and laptop, but I've seen a lot of talk of hosting on GitHub or similar services as well, which would give you revision history and authoring if you needed to know who did what

I do a tiny bit, but even less structured than you're describing (and less consistently), so I'm curious to see other responses here.

I use org-journal[0] in Emacs as a very efficient lab notebook. I use the Doom[1] configuration, so this is my workflow:

- launch Emacs

- type `Space n j j`, which opens a journal file customized the way I like it*, inserts a date heading and timestamp, and sets me in insert mode

- type my timestamped notes

I can do this from any buffer in Emacs, so it's really convenient to stop in the middle of something, jot down a note, and then go right back to what I was doing. I develop iOS/macOS software right now, so the switch to Emacs from Xcode is a little more friction than I used to have, but it's so useful I don't mind it at all.

* I have a weekly journal in a directory for the year, titled week number-month-day that started that week (this week's is `34_08-23`)

[0]https://github.com/bastibe/org-journal [1]https://github.com/hlissner/doom-emacs

Very similar to the way I use it. Ctrl-c n j makes a new entry in org journal, carrying over any TODO/PROG items from previous days if the current day file doesn't exist. I use it all the time and it's replaced my beloved hardback A4 graph paper notebook.

I keep my journal files hidden from the Treemacs sidebar so it doesn't get cluttered.

I keep a permanent in progress item with sprints and their start/finish dates, backlog items, etc etc.

Does anyone here keep a log or journal at work? I'm leaving a place that I've been at for quite a few a years and I am a little regretful for not writing down some observations and predictions as I had them. It seems like nothing the business did over the long term went according to plan, but I've lost a lot of the past perspective so its hard to compare.

I keep a journal, because otherwise I tend to forget or minimize much of what I've done. It helps me remember the big things when it comes time for performance reviews, and it also helps me remember the small things with accurate sprint retrospectives. Spending several hours per week fighting with environments, retrofitting code to ancient dependencies, and investigating build failures that turn out to be local to one person's laptop are time sucks that need to be called out in retrospectives so that people understand the potential value of paying down technical debt. Those things are also exactly what I tend to forget about by the end of the week.

My journal consists of a notebook with a new note for each day, titled with the date for easy sorting. I copy/paste in a template that looks like this:

    Morning checklist:
    _ Check email
    _ Check Slack
    _ Make sure my tickets are in the correct state
    _ Look at the meetings on my calendar and decided how to prepare (or not) for each one
    (etc.)
    
    Plan today:
    
    Actual today:

I love the morning checklist because it ensures that if I space on one of those items during the day, like updating a ticket, it'll get taken care of the next morning. Trusting this part of the process takes a lot of stress out of the rest of the day.

After I make my plan for the day, I don't touch that section again. That way I can compare and contrast my plan with what actually happened. It can be enlightening to look back and see that a development project was in my plan every morning for a week, and my "actual" was dominated by production support and fighting with test environments.

The best part is that since I started keeping a journal like this, I've realized I get way more done than I give myself credit for.

When I was working for scientists this was a habit I learned and loved it, but due to some other factors I became aware that in more traditional business the kind of frank statements I put in logbooks could be a potential issue, and have found myself not doing it due to the chilling effect.

Yes, religiously. I have a list of tasks for each year which links to a file with my notes on each task, as well as a daily journal where i log my work hours, random notes from meetings, etc. It is exceedingly helpful.

I did. It was valuable to be able to go back and refer to discussions about the architecture of something, or to note down obscure details. If you choose to do it on paper, make sure you photocopy regularly.

I do. I have four journaling system.

1. Small Whiteboard - Daily Todo & quick notes for me to remember later. Easily erasable and save paper when it is not need. It is always on my right-side corner of my desk for easy access and quick jot note. And I take a picture of the small whiteboard at the EOB for archival reason in case if I need to go back.

2. Legal Pad - For weeklong Todo, project notes (for developing forms and contracts), clients, etc. I have two pads: the normal pad and the small pad (5x8). I have more than 10 small pads for various functions since they are small and don't take much desk space. They are easily accessible when I can grab the pad quickly and write it down.

3. Obsidian - For documentations and information for long term storage, including development planning and various stuff.

4. Google Docs - For collaboration Todo, agenda for the meeting, conversation notes, minutes, etc. to share with my boss. I could use Obsidian for collaboration but my boss is not knowledgeable with Obsidian or any journaling software and it will be pain for him to set it up. I rather to make it simpler for my boss and easy access for him to look into.

I strongly used 1 and 2 because I found that jotting it down via handwriting is quicker and ready to use than doing it electronically. Doing it electronically can take some time to get into typing mode when time is critical like in the middle of the phone call. Pens, papers & highlighters are always available and ready to use within a second.

I have a ‘check spindle holder’ for my small legal pad paper when I am done with it. I just “whamp” the paper on it and get the satisfaction and stress relieving from it.

Yes, for every important long term goal/task I'm working on. Eg. I have a journal file for feature foo I'm working on and another one for bug bar I'm looking into, but I don't immediately create one for something I can do in a day or two.

I do, yes, and I ask my team to do the same. Even in software development it's very helpful. It helps the team stay in sync more, reducing the need to meet. It helps me stay in sync from yesterday me to today me as well. It avoids duplicated work as well. I consider doing this part of professionalism 101 at this point.

Kind of. As I mentioned elsewhere (https://news.ycombinator.com/item?id=28327689) I recently read a very interesting post on Xe’s blog and that inspired me to start writing things down in a physical notebook.

I think that rednotebook is invaluable for this purpose.

It's a nice linux app that works like.. a notebook.

yeh I keep digital notes, a notebook and a load of sticky notes. Theres no real system to it (just the way I like it), but generally speaking working notes go in the notebook (also for tangent tasks), important reminders on the sticky notes (pasted all over my monitors) and "knowledge" goes in the digital notes. In reality its a lot more sporadic than that, but I know where to find things

A lot of the goals/problems mentioned in the document can be solved with a "lab notebook" consisting of plain text files plus Git version control. For example:

> Legal document to prove patents and defend your data against accusations of fraud

> Bound/Stitched | No lost pages, legally stronger | Difficult to copy, not logically organized

> Requires electronic security, corrupted files, software compatibility issues

> No pages come out of the notebook | Do not take any pages out or remove any data | Do not skip pages in your notebook | Cross out any unused parts of a page

> Correct mistakes, do not remove them | Cross out mistakes with a single line | Paste in corrections without covering anything | Sign and date all corrections

Also, if you sign your commit hashes with a trusted timestamping service, then that could serve as watertight evidence of prior work.

Programmers really love plaintext/markdown notebooks, but they introduce a lot of friction around graphs and "snap a picture + annotate," and that destroys their value prop for me.

Snap + annotate (or sketch + annotate) is an absolutely killer workflow if you deal with artifacts in the physical world, to the point where I gladly endure poor text editing capability and version control so long as I can have snap + annotate. Heck, if I didn't have a computer with good snap + annotate support I'd use a physical lab notebook before I tried to somehow shoehorn the workflow into text.

Reminds me of this post from June, which inspired me a lot: https://news.ycombinator.com/item?id=27639875

I have bounced around between different lab notebooks, obviously starting with paper. I then moved to a word doc. Problem with word documents is that they don't have any good hierarchy. I then tried Obsidian, which is a markdown editor. It's nice, but the table and image support is lacking. I need to be able to paste in tables from a spreadsheet because that's how I set up many experiments.

I've always been attracted to the idea of keeping a lab notebook for my work, but since I do engineering/scientific computing, the concepts never seem to directly translate. Does anyone have any resources for keeping a similarly structured log of research/work done on non hard science projects?

If you're in software engineering, Jupyter notebooks support a large variety of languages now.

I keep handwritten notes in a notebook of recipes I particularly like. These are useful for making adjustments over many preparations. I guess the bottom line is being willing to enforce an organization on your notes, and then iterate towards a format that makes sense to you. For the recipe notes, time series (like journal entries) works well. Or inline. It depends on the note. Ingredient substitutions can be time series, but wrong measures specified in the recipe would be an inline correction.

> engineering/scientific computing

Definitely Jupyter notebooks or other "literate computing" methods. Emacs org-mode has literate programming stuff, but honestly Jupyter is easier to use.

Thanks - I'm not a software engineer, but I write a lot of code anyway. I've used Jupyter notebooks to good effect in the past for smaller, self-contained projects, but they don't quite seem to fit my workflow well enough for me to use them for everything. You make a good point about picking something and enforcing it - I just have a hard time even finding something to start iterating on. Maybe I just need to be more creative about how I think about/use Jupyter notebooks...

> something and start iterating

Go with a standard time-series journal. This is about the simplest way to do things. It works well enough for me with minor additions.

Eventually you've referred back to your journals enough to know what kinds of information you'll be looking for. I've gotten part way there, so far. I'm still trying to sort out details for a lot of stuff; where does the plan of a project live? In the journal? In the projects files? What about the day-to-day entries? What about troubleshooting epilogs? Like I said... I'm still trying to sort out where each of these live.

I'm tempted to just throw all the things in project folders (because they're all projects anyway) and then grep for stuff if I need to find it.

But putting everything in one folder still doesn't satisfy my itch to see my journal from two views: time-series (which answers "what did I do X days ago?") and project collections (which answers questions like "what's next?" and "how did I solve this obscure problem?!").

You can use use Jupyter Notebooks like Markdown files, with the added benefit of code blocks being executable for reproducible work.

Then, just throw your notebooks into MkDocs (using a plugin) and voila, you get a statically hostable version of your notebooks!

Easily coupled with Git and CI/CD practices, you have a nice, easy to use setup for prodyctive work and knowledge sharing if needed!

> Most importantly, your lab notebook is not yours...You should not take it home for any reason.

Dafuq?

How I take notes: For professional life: Pen + dot grid notepad - developed independently, but similar to the bullet journal technique. Dot grid plus pens make the whole thing ultra customizable. I can sketch engineering designs, make a calendar, track action items, take detailed notes, all in the same format. The key is to be strict with page numbers, dates, and index as much as possible.

Things I occasionally miss - keyword search (I can still look things up by date or subject in the index), multimedia inserts (think dragging video/photos/sound clips into one note), never ending space (notebooks run out of pages), easy backups (thinking about digitizing with photos or scans), team collaboration (if this is necessary I use Trello).

Things I like - no OS/tech stack compatibility issues, "it just works", lighter then a laptop/tablet, don't need to charge, easy to read, can bring into a secure area (where outside electronics are not permitted), travels well, hard to damage.

http://bulletjournal.com/get-started/

For personal life: add Google keep for simple lists, and then a mix of Trello and dot grid for larger projects (less strict formatting than professional life project management).

How I index using pen/paper:

Every page has a number in the upper outside corner.

Every entry into the note book has a title + subject tag, date, and the initials of a list of people linked to that entry. For easy reading and scanning I put all of this on one line. The title and subject are dark lined (thick), followed by the date MM/DD/YY, and then the initials inside a ( ). Each piece of info is separated by a ";". All of the info listed is underlined.

under the intro line, I also break down actions, notes, calendar adds, etc with 3-4 different symbols. All I all this makes it clear and easy to scan quickly as you are looking for things.

Each month I have 4 or 5 pages devoted to organization.

I have a task list. Two bulleted columns. Old tasked are migrated over from the previous month. New tasks are added as soon as they are generated. Completed tasks get an X through the bullet. Migrated tasks get an >. Tasks that are linked to journal entries get a page number and a tag.

I have a calendar page. 1 column, all dates of the month listed . Meetings, important actions and events are are listed by day, or with arrows over multiple days. Linked by tag/page number if applicable.

I have a subject index. Major tags, reoccurring subjects, get a line or two. Page numbers with those subjects are added to the index.

I realize this sounds complicated, but it's not once you get into a routine. The biggest this is it's okay to make "mistakes", and to get used to not having a delete key. Change is part of the notebook. Think of it like a change log. It's also important to be able to see what you change. Since I work in pen, I just make a single line through things I want to change.

Note: when I finish or complete something on a task list or the calendar, I'll mark or change the symbol, I won't cross something out just because I'm done with it.

Finally, I have a bunch of multi color pens I use. They are all the same brand/model, but I tend to use a different color each day. That makes it obvious as you are scanning that you are on the next day. I don't use specific colors for anything in particular, too hard to keep track of.

Hope this helps.

Reposted from an older thread @ https://news.ycombinator.com/item?id=17799232

edit - notebook choice is important. I use something like this (https://www.amazon.com/Dotted-Bullet-Notebook-Journal-Hardco...) Not an affiliate link. Its sturdy, you can't pull out or add pages (which is a feature), bound and stitched, decent quality paper, dot/grid, and normally under $15/book.

I've done something similar for decades. I work in science, and need to cross-reference a lot. I have a simple scheme for that.

When I refer to another page, I write that page number in square brackets (in the text, and repeated in the outside margin). Reference to other books have the same, but the number is prefixed by an index number for the book, and then a colon.

The page referred to also gets an entry added, in the same format but in angled brackets.

This makes it really easy to propagate new information through my work.

Apart from the possible addition of the angle-bracket entries, I never alter existing text. If I redo an analysis, I simply make a wholly new entry, and add an angle-bracket notation on the old entry.

The square-bracket and angle-bracket scheme makes it pretty easy to update analyses, e.g. if instruments are discovered to be faulty on a certain date.

In addition to this referencing scheme, I also have two-letter keywords for topic areas, and those go into the table of contents.

Another difference is that my dates are written yyyy-mm-dd.

A policy of photocopying new work every Friday afternoon is quite sensible. I don't keep the copies with the original work, though, out of fire/flood/etc concerns.

Oh, one more thing: about once a week, I type the item titles into a text file, along with book number, page number, keyword list and both square- and angle-bracket references. That way I can use `grep` to find things. I suppose I could write a few lines of code to auto-generate a local webpage, but I'm just as happy listing the file or looking in a text editor.

"We are living in an era of the most sophisticated technological advances possible, and yet the treatment of cancer is paleolithic." — Azra Raza

Reply on news.ycombinator.com