Hacker News Reader

Comment by neves | Hacker News Reader

I just find reproducible notebooks at the internet. It is really rare to find them from coworkers. If they aren't trained as developers, it is almost impossible. Their solution for this problem looks really efficient and is really simple and brilliant:

> Writing Polynote’s code interpretation from scratch allowed us to do away with this global, mutable state. By keeping track of the variables defined in each cell, Polynote constructs the input state for a given cell based on the cells that have run above it. Making the position of a cell important in its execution semantics enforces the principle of least surprise, allowing users to read the notebook from top to bottom. It ensures reproducibility by making it far more likely that running the notebook sequentially will work.

View on HN

Some related links if you’re interested in this kind of approach:

https://nbviewer.jupyter.org/github/friggeri/notebooks/blob/...

https://github.com/jupytercalpoly/reactivepy

https://dataflownb.github.io/

https://github.com/stitchfix/nodebook

Thanks for the kind feedback. It's a young project to be honest, but I'm pretty proud of what we've done with only two contributors so far. With community participation I think we could support many more languages pretty quickly!

I really have always wished for reproducibility. Thanks for taking up this feature. How do you handle aliasing and references inside objects? Suppose I have

    #Cell 1
    a = [1,2,3]
    b = (a,True)

    #Cell 2
    b[0][0] = 5

    #Cell 3
    print(sum(a))

Now if I change Cell 2 to

   # Cell 2'
   b[0][0] = 4

and execute, Cell 3's result becomes stale. Do you track such dependencies? Would really love to read more about the underlying implementation.

If you mutate an object itself, we can't really track that. There's no magic going on; you can break the state if you use mutable objects. It's less of an issue in Scala where immutable data structures are the norm, but I can imagine it would be disappointing in Python.

Currently it takes a shallow copy of the state output by each cell, meaning every value is going to be a primitive value or a reference. If it's a reference to mutable state, you're kind of on your own with respect to keeping reproducibility. I felt like this was a good compromise between strictly enforced reproducibility and practicality; if it turns out to be confusing we could consider deep copying the state, or having an option to do that (I could imagine it being pretty bad for efficiency in a lot of ML use cases, though).

I am not familiar with those notenooks. What would be wrong with re-executing all the cells below the one that changed?

That is usually a feature. The reason it's not the default everytime you change a cell if that cells can contain long running calculations.

You could also try Voilà to make reproducible standalone app from notebook https://blog.jupyter.org/and-voilà-f6a2c08a4a93

Reply on news.ycombinator.com