JupyterLab: the next generation of the Jupyter Notebook

The %Rpush and %Rpull magics are what you need.

Also, if you are using Spark, then Apache Toree[1] lets you use Python, R, Scala and SQL in the same notebook against Spark[2].

[1] https://toree.incubator.apache.org/documentation/user/how-it...

[2] https://github.com/ibm-et/spark-kernel/wiki/Language-Support...

For other plugins there is very nice integration so that you can have interactions in one plugin update outputs in another and vice-versa.

The ease of creating a connected ecosystem of web-applications connected to your workflow is one of the powerful aspects of the new platform.

What would great integration look like to you?

I like super boring static charts, such that I can completely decouple the presentation from whatever business API was used behind the scenes for the data that's displayed. Take a super severe Occam's Razor approach to absolutely avoiding any kind of animation or interactive plot for as long as possible, and only if there is some kind of outrageously severe business case that absolutely demands it (which there almost never is) will I resort to designing a way to productionize the widget aspect of the report visualization as well.

For this reason, I now see widgets like from IPython as a huge minefield. It looks pretty, but it's a big trap, and it doesn't add nearly the value everyone thinks it does.

(Just to be clear, I'm only talking about the non-demoware case, which is the kind of case I have always worked with.)

There is seldom any value in looking directly at code and at a console at the same time. But if you really want that, it's super easy to do it with a window manager like xmonad, or even just arranging shell windows on your desktop so that you can alt-tab between them easily.

You often want to quickly spawn and kill shell tabs, which themselves may or may not be in the same language. For example, I often have a tab in which I'm using IPython, another tab in which it's the working directory so that I can execute things with Python directly, mv/cp files, ls, etc. And then still more tabs in which I have background processes that check for file changes and run my unit tests whenever things change, sometimes a tab for Python 2 and another tab for Python 3. And further, development projects are almost always cross-language, so I tend to also have some tab opened for writing and working with C or Haskell at the same time, and possibly another with a psql shell.

Since there are so many necessary tabs just to do even the tiniest things, it means that any and all visual overhead must die. Switching these tabs, even if it is quick, inside of a clunky GUI application like a browser is just too unproductive -- the browser periphery already wastes maybe 5% of the available visual space, and then the overhead for the tab icons, clickable close buttons, etc., wastes another 5% inside of that, and then the width of the tabs is restricted because there's some left panel with directory information or in-memory workspace information (both total wastes of time), and the height is restricted below by some worse-than-plain-shell console; it just makes no sense. It's too much visual clutter and too inefficient to facilitate switching around as much as is necessary.

I think it should be emphasized again that the left-side panel showing either directory structure or the contents of an in-memory working environment are huge wastes of time. If you need to visualize a directory structure, that should just be another buffer, like a code source file, and when you want to view that you just switch to that buffer. There's no benefit to having some of your visual field distracted by it when you're working on other files. And the in-memory information is also generally crappy. It's another thing where if you really need it then it should just be in another buffer and you should quickly go to that buffer and give it all your attention for the short time you need it, then go back. It serves nothing by having it as an ever-present visual distraction. But more than that, relying on inspecting variables that way is a very infantile thing, and I see it a lot with MATLAB programmers. Their form of debugging is not to scientifically inspect and control the execution of the code, and use proper breakpoints and watchpoints to tells them what's going on, but instead to just "run everything" and then go and click to open up a spreadsheet-like view of a matrix variable or something and manually (!) inspect the data. Then they become reliant on this as a crutch and complain when it's no longer there, instead of learning proper ways to write tests and proper debugger usage and let those things automate the problem of zooming in on outlier data, messy data, or bugs.

Anyway, there's plenty more to say, but it's probably long-winded enough.

To me, the traditional IDEs do work well for debugging and software development.

Notebooks are great for explanatory examples and interactive experiments. I think these are different to the type of software development I do when I use an IDE.

For example, I find notebooks great for rapid iteration of parameters when I'm doing "data science", or indeed most of the feature extraction->modelling->prediction data science pipeline.

What I don't find them good for is developing new algorithms. It isn't clear to me if this is an inherit limitation of the notebook format, or just something where it needs new developments.

(To be clear, I've also used both Zeppelin and Beaker notebooks and don't see any particular advantages. I've also used R Studio, but I don't really know enough R to comment sensibly on that)

As someone who still reaches for a Smalltalk environment when I need to prototype something, ipython notebook is about the closest thing I've ever found to the style of interaction you get with a Smalltalk REPL (aka "workspace"). Smalltalk deliberately blurs the lines between a text editor and REPL, and the debugger takes this a step further - a combined editor and REPL, within a suspended execution context. It could be argued, for example, that coding within the suspended context of a failed unit test, while the code underneath your cursor has this REPL-like liveness, is the non-cargo-cult way of doing TDD.

I'm not trying to claim Smalltalk as the Greatest Thing Ever, but its existence (and its "otherness" - from the point of view of today's conventional style of development) are evidence that there are useful tools to be had, somewhere down a road less travelled.

I've actually been working on implementing something to that effect for Python (see http://kitaev.io/xdbg). But it's been really hard to figure out who already has experience with similar workflows, because these tend to be isolated to particular language or devtools communities.

Apparently I need to take a closer look at Smalltalk.

It looks nice, but the installation experience is (was?) terrible (as in - didn't work at all). Note the long gap between the 0.5.6 release (January) and the 0.6.0 release (July)? There were 3 (4?) Spark releases in that time, and that meant that none of the out-of-the-box released worked for anything except the version of Spark you downloaded with it (and from memory that had problems too)

I got it working and evaluated it in some depth. I'm from a Java background, so I really wanted to like it.

But it turns out that all those features that seem really nice are mostly only nice if you are trying to build applications, not notebooks. Maybe it has improved, and maybe for some usecases it makes sense.

Currently, you can easily control which things are saved and which aren't. So it is the best of both worlds.

As late as 2000, this was the single biggest advantage and single biggest impediment to new programmers in MIT's 6.001 lab: a bunch of nonvisible state, mutated by every C-x C-e. The student has tweaked two dozen points trying to fix a small program, and re-evaluated definitions after many of them, but maybe not all. The most straightforward help from a teacher is to get the buffer into a form such that M-x eval-region paves over all that, sets a known environment of top level definitions, and---more than half the time---the student's code now works.

I have similar concerns about much of Victor's work, for the same reason. Managing a mental model of complex state is a n important skill for programming, but it's best learned incrementally over long experience with more complex programs. These very interactive environments front load the need for that skill without giving any obvious structure for helping the student learn.

Contrast Excel and HyperCard, which have no invisible state: you can click and see everything.

I still rely heavily on Mathematica, since the notebook interface won me over years and years ago. I think the next Mathematica paradigm to be redone in JupyterLab is easy connections to data without all the connection string fuss. You can just put in some basic phrases in Mathematica, and you are connected to the data, albeit curated by Wolfram, but still simple and immediate.

I can link to Mathematica from NetLogo and other programs I use too.