Hacker News Reader

Comment by pletnes | Hacker News Reader

Sometimes I work on software development, and this mindset («the only valuable asset is the code») makes total sense. But if I work on analytics / datascience projects, the analysis including outputs could be time consuming to run, validate, and visualize. In these cases, it might be required to version the outputs.

I’ve never used jupyter for taking notes in a lab setting, but with more and more instruments being computer/network connected, I imagine this would make total sense - put your data and notes with your analytics work.

Many jupyter users are not «software developers», they just use code to perform their work.

View on HN

Precisely this. When your output is something like research data, or even just something that generally takes a long human time to complete (hours vs Microseconds) it makes a lot of sense to version and keep outputs, at least on major "versions".

But would you version it by storing it as output in an ipnyb file where it is overwritten if you rerun that cell? I would store the data in a versioned database or as separate data files in the repository (possibly stored in git-lfs). And I would store results of the analysis as data files / image files / whatever else, NOT as ephemereal outputs in an ipynb file. But I am pretty far down the “ipynb files are for local use only” path.

Yeah if your analysis to takes hours to run, you should really split up the number crunching code and result analysis / visualization. Not only does it make version control of the code easier, you can save the output in an organized labeled manner (time-stamped, etc.) and, if you lose power or the kernel crashes, you don’t need to rerun the lengthy analysis if you want to make a change further down the pipeline.

It was an extreme example to drive home the point that one is "human scale time" and one is "computer scale time", people are reading far far too much into my choice of hours specifically there.

However, I wouldn't then use version control software like Git for versioning analysis objects, as it is designed for text file source control and diffs.

(How one does a diff of a data object look like? If there is a natural text format to save it in, it still is usually quite messy, and Git doesn't really like Gb sized csvs.)

My preferred workflow is to version the source files in Git and store the associated data objects in a separate archive directory with meaningful name and the hash of commit of generating code as metadata attribute.

Now if you had a version control "IDE" software that would render changes in figures and other blobs nicely, then it would make sense to build a workflow around it.

What’s your solution for hosting the archive of the data if you want to share it? That’s a weakness in my workflow.

I use nbdime and have integrated it into my git diff so I see graphical diffs. Github renders notebooks with their output intact. They recently announced [rich notebook diffs](https://github.blog/changelog/2023-03-01-feature-preview-ric...) as well.

Simply write the outputs to a formatted file and keep it separately?

Reply on news.ycombinator.com