Hacker News Reader

Comment by nl | Hacker News Reader

Context: We use Jupyter heavily (mostly against Spark).

In my experience there is a set of things that "traditional" Jupyter notebooks does really well. Anytime you have a linear flow of steps the notebook metaphor works really well.

However, if you are doing things approaching traditional development, where you have multiple sources of data, or loops that require debugging, or basically anything that isn't linear in nature it doesn't work so well.

I wouldn't want to lose traditional notebooks, but I'd love to be able to offer people something like this that offers better debugging and some development tool support, rather than jumping to a full desktop IDE.

View on HN

My experience is in line with yours, debugging loops and functions is a big pain point.

However, I think there's a much better solution to be had here, which is to add more powerful debugging capabilities to Notebook. I think Notebook has potential for new debugging paradigms, imagine for example being able to break anywhere in a cell and get a new 'forked cell' which operates in the context of the code that you just broke into. I think that's the better direction to go instead of reverting to the very interfaces and paradigms that we moved away from.

I'm not sure (And by that I don't mean I disagree: I'm genuinely unsure).

To me, the traditional IDEs do work well for debugging and software development.

Notebooks are great for explanatory examples and interactive experiments. I think these are different to the type of software development I do when I use an IDE.

For example, I find notebooks great for rapid iteration of parameters when I'm doing "data science", or indeed most of the feature extraction->modelling->prediction data science pipeline.

What I don't find them good for is developing new algorithms. It isn't clear to me if this is an inherit limitation of the notebook format, or just something where it needs new developments.

(To be clear, I've also used both Zeppelin and Beaker notebooks and don't see any particular advantages. I've also used R Studio, but I don't really know enough R to comment sensibly on that)

This sounds like a promising idea.

As someone who still reaches for a Smalltalk environment when I need to prototype something, ipython notebook is about the closest thing I've ever found to the style of interaction you get with a Smalltalk REPL (aka "workspace"). Smalltalk deliberately blurs the lines between a text editor and REPL, and the debugger takes this a step further - a combined editor and REPL, within a suspended execution context. It could be argued, for example, that coding within the suspended context of a failed unit test, while the code underneath your cursor has this REPL-like liveness, is the non-cargo-cult way of doing TDD.

I'm not trying to claim Smalltalk as the Greatest Thing Ever, but its existence (and its "otherness" - from the point of view of today's conventional style of development) are evidence that there are useful tools to be had, somewhere down a road less travelled.

I learned enough Smalltalk to make prototypes with Seaside. I love the way notebooks work and to make them more Smalltalky would be wonderful not only for programmers but for the field as a whole.

I agree, the notebook could really have much more powerful debugging.

I've actually been working on implementing something to that effect for Python (see http://kitaev.io/xdbg). But it's been really hard to figure out who already has experience with similar workflows, because these tend to be isolated to particular language or devtools communities.

Apparently I need to take a closer look at Smalltalk.

Seeing as you're heavily using Spark, have you had a look at Apache Zeppelin (site: https://zeppelin.apache.org, demo: https://www.youtube.com/watch?v=J6Ei1RMG5Xo)? Seems like a more powerful notebook approach, plus better architecture for using embedded d3.js viz. Also painless templated SQL -> published dashboard looks great for getting data visible early on.

Yes. Not really a fan.

It looks nice, but the installation experience is (was?) terrible (as in - didn't work at all). Note the long gap between the 0.5.6 release (January) and the 0.6.0 release (July)? There were 3 (4?) Spark releases in that time, and that meant that none of the out-of-the-box released worked for anything except the version of Spark you downloaded with it (and from memory that had problems too)

I got it working and evaluated it in some depth. I'm from a Java background, so I really wanted to like it.

But it turns out that all those features that seem really nice are mostly only nice if you are trying to build applications, not notebooks. Maybe it has improved, and maybe for some usecases it makes sense.

One of the Apache Zeppelin committers here.

Thanks for honest feedback (although it sounds bitter) - we are working hard on improving situation with release schedule indeed.

If there is any chance you could elaborate on installation experience though a JIRA issue [1] - that would be very much appreciated.

1. https://issues.apache.org/jira/browse/ZEPPELIN

Jupyter really is just for analysis. I prefer standard IDEs like Jetbrains for application development.

Reply on news.ycombinator.com