Hacker News Reader

Comment by kingkongjaffa | Hacker News Reader

I don't mean to be a philistine but just use an excel sheet, if you are super anal use onenote/evernote.

I've looked into electronic notebook software, and kind of started out with a similar view. What I've learned is that there are two ways of thinking about it:

1. Replacing your personal notebook with an app on a computer. That's where things like Excel, Jupyter (what I use), or EverNote, could work for you. It doesn't seem like there's much for a science-specific app to offer here.

2. Then there's electronic lab management, where you're not managing the entry of data so much as managing the people who work in the lab. This is where the software developers are active, with a variety of ideas for what lab management consists of, that probably vary depending on your discipline and industry. For instance, the website mentions protocols, collaboration, and security, none of which are issues in my work, but are very much issues in a large managed research group that is potentially in a regulated or litigous industry.

The first option works just fine for me, because it addresses the main feature of my work: Almost all of my experiments are automated, my data are produced in computer readable format, and all of my analysis is done with software. So I pretty much have a computer handy whenever I'm doing "lab" work. It would be a different story if I were, for instance, doing field or wet lab work. Then I'd probably stick with a paper notebook.

What format do you use? I was searching for scientific notebook templates and examples but didn't find any so I just made my own: pandoc tags on the top and the rest is GitHub Markdown. Python cells when data need to be processed.

I use vanilla Jupyter as bundled with the WinPython distro. Don't ask me why I went with WinPython instead of Anaconda, but now it's what I use.

My typical notebook is mostly Python with a bit of Markdown, rather than vice versa. I'm rarely creating a linear narrative, so I often go back and add explanatory text after the fact but before I forget.

It's worth noting that I'm a lone wolf in terms of maintaining my own lab data and results. Nobody else cares how I do it, and I'm not in a regulated field. I'm in industry, so I don't publish. So it's mostly so I can figure out later what I did. At home, I've put some really obscure Jupyter notebooks on my GitHub page, which so far appears to have received exactly zero views. GitHub renders Jupyter notebooks automatically.

Alongside Jupyter, I've often got a regular Python IDE running, with programs or scripts that I use for data gathering. Most of my lab equipment is accessible via pySerial (a requirement for anything that I buy new). If you're really careful, you can script your data collection within the same Jupyter notebook that you use for analysis, but there are some hidden dangers, such as losing data stored in variables when you terminate the kernel, or over-writing files. My habit with data files is to use str(time.tim()) for file naming, so it's impossible to overwrite a file by accident.

In my opinion, Excel is not a good platform for documenting research because it is very time consuming to audit and try to reproduce the results since there is no standard way for inputting data.

There is a long list of cases where companies and governments have made huge decisions that were based on Excel spreadsheets where the models had fundamental flaws [0][1][2].

[0] https://www.forbes.com/sites/timworstall/2013/02/13/microsof...

[1] https://storagemojo.com/2016/08/26/excel-may-be-dangerous-to...

[2] http://www.jamaicaobserver.com/business/Excel-is-dangerous-_...

>There is a long list of cases where companies and governments have made huge decisions that were based on Excel spreadsheets where the models had fundamental flaws

Sure, but how do they compare with the (IMHO countless) number of "right huge decisions" made on actually accurate and well devised Excel spreadsheets?

Excel is a tool, it has its limits like any tool, and should be used - like any other tool - with caution when "huge" decisions are made on it.

And like any tool you cannot put the blame on it because someone cannot use it properly.

No, Excel is to blame, because it's way too happy to convert your data into something else it thinks is appropriate. A proper system (SPSS, R, Pandas, you name it) will yell at you if you try to do nonsensical stuff.

No doubt that there are "better" tools, still you can't put the blame on Excel. You can choose to put it on either the sloppyness/lazyness of not having checked (double checked) data and formulas or to the decision to use Excel (possibly due the ignorance of the existence of other "better" tools).

Anytime someone says excel I immediately think that they would be better off with a real database. I'm rarely wrong.

This is defiantly a case where you want a database. You want all data to go into a write-only table (when you make a mistake the correction needs to be noted and tracked). Then use queries to do your analysis on the raw data assured that you won't make a mistake.

A database also does well for collecting meta-data: who entered this data when.

Finally a database is very useful for meta-analysis which is starting to become important.

Or, you know, an actual notebook.

Yes

Nothing is easier than using real paper and a pen.

But as kabdib has said many times, "You can't grep paper."

Yes. But more and more, you can easily photograph and OCR it.

unless there are computational aspects to your work, which these days there usually are

But all of these ELNs are targeting replacing wet lab notebooks. I tried many ELNs and even wrote one, but I don't think there is any replacing paper notebooks for wet lab research. Even when you have some computational aspect, you can print out a figure or table and attach it to a page in the notebook.

(Now reproducible dry lab notebooks are a whole other matter)

I run a computational lab, and I still use a paper notebook

I did wet and dry work in my PhD and before, and found a paper notebook to be incredibly inconvenient. Finding things, referring to things, including data or code - all very laborious if it's even effective. I can imagine keeping thoughts or notes like this, but not my lab book.

I use monthly markdown files where I can link out to code, commits, issues, include figures etc.

I found the opposite to be true - electronic lab notebooks were almost immediately abandoned, or in one case turned into an illegible wall of madness.

To each their own I suppose.

Electronic Lab Notebooks are quite often used in industries where there's a regulatory requirement to record who performed an experiment, and to digitally sign what the results were. For simple research Excel is probably fine - but this software is targeted more at the incumbent software (which isn't particularly intuitive IMO).

None of the above are FOSS though.

Instead, I would suggest org-mode or maybe cherrytree.

Reply on news.ycombinator.com