Hacker News Reader

Comment by alan-hn | Hacker News Reader

I've never been a fan of notebooks, it always felt like a strange way to run my code. What's the real advantage over creating scripts? If I want my code to run somewhere I would need to notebook architecture or just turn it into a normal script anyhow

View on HN

Notebooks have at least one advantage, but it's one that shouldn't really be an advantage: In a world where every big organisation rushed to adopt all the latest cloud buzzwords, we now have an awful lot of half-baked fontends for AWS and Azure that will certainly be mandated by company policy. Actually getting a computer that can access all the data you want and run in a way you want through EU law, US law, EULAs, company security policy, cloud billing policy et cetera et cetera is a nightmare.

But for some reason, no matter what nonsensical environment you're being forced to operate in, there's always a Jupyter web interface with access to exactly what you need that none of the auditors, accountants or systems administrators seem to care about. It may not be good, but it works, and won't be bricked by some inexplicable change to IAM roles or VPS settings applied by some automated policy tool you didn't even know exists.

Same reason spreadsheets are so powerful.

For a script, you run it from start to end.

For a notebook/repl environment, you can create any number of intermediate steps, rerun the previous step with minor modifications and check if the results are better, rinse and repeat. For jupyter notebook specifically, you can visualize data and add markdown inline which are very useful.

You won't understand it unless you are already familiar with the workflow.

Sometimes I need to run my code in small pieces for testing and evaluation, I do this with multiple smaller scripts. Data can be saved to files and this accomplishes nearly the same thing as a notebook without needing to have the notebook environment

Sure, if you don't mind the overhead (both development and processing time) of loading/saving state to disk (about 10-30 minutes for a lot of my data). In notebooks you don't have to think about it since it's just the objects in memory (and indeed they don't make it easy to think about it, which is a reasonable criticism, but I don't currently know of a framework or system which gives you the advatages of both).

Whenever I work with large datasets I use a small subset of the overall data to do testing while I build the pipeline, this avoids long run times and allows for quick iteration while I get things set up to run against the full dataset

of course, running a script from start to finish is better when it's fast, but if your code is slow to run, each cell acts like an in memory cache of your previous work.

There's so many other ways to achieve the same effect though. I would argue that serializing the result of some expensive computation explicitly is even just better, as it can have a meaningful name and metadata.

Just serialising and deserialising the results can be expensive though: it's quite for jupyter notebooks to have >64GB of data loaded. Even with high-end SSDs that's a fairly significant overhead.

I've seen them used really effectively for debugging and incident response & documentation in elixir and erlang. You can connect to a running node in a controlled way, repl around the live environment, and dump potentially interesting things out to the notebook as you build your understanding of what's going on.

I wasn't too active in ops stuff at that job, but as a dev it was so awesome for them to hand me off a notebook that was basically an in-progress investigation, including the entire path of exploration they took to narrow it down and dead ends along the way. You can't ask for a better bugfix ticket.

Now I guess a lot of that depends BEAM particulars, and doesn't apply to jupyter or whatever. But it's at least an illustration that notebooks can uncover novel workflows for certain tasks.

Reply on news.ycombinator.com