jbednar

I've added mesop to https://pyviz.org/tools.html#dashboarding, where it is the 35th OSS Python dashboarding tool to appear. (There were 5 in 2019). It's a crowded space!

https://holoviews.org is probably a better link.

Panel, hvPlot, HoloViews, GeoViews, Datashader, Param, Colorcet -- all working together to make Python data visualization easier and more powerful.

You can if you use Panel (panel.pyviz.org); it does not depend on Jupyter in any way (but like Voila, works well in Jupyter).

Panel and Voila attacked the same problem (moving easily between Jupyter and standalone server contexts) from completely opposite directions. This difference has some implications on their design and function.

Voila is based on ipywidgets running in Jupyter notebooks, and to make a standalone dashboard they had to create a standalone server that can securely execute Jupyter cells and display the results without allowing arbitrary code execution. The server is thus a work in progress, while the Jupyter integration was already solid.

Panel is based on Bokeh models, and because Bokeh models already had a full-blown standalone server, the task for Panel was to make Bokeh models (a) work seamlessly in Jupyter (previously they were awkward and limited in that context), (b) support other plotting libraries (by wrapping everything as a Bokeh model), and (c) have an API that's easier to use than native Bokeh for easy prototyping and design. Solid, secure server support came for free.

Once both libraries support each others models (soon!) and get a bit more polished, then off the top of my head, the main differences will be:

- Panel can use a Jupyter notebook, but it works equally well with a plain Python file; the notebook is just a source of Python code for it. Panel can be used fully even without Jupyter installed. Voila is closely tied to the Jupyter cell-based execution model, which is good or bad depending on your point of view. - Panel allows you to construct a "server view" of your notebook that can be completely independent of what is shown in the notebook, even though it is specified inside the notebook. I use that capability to have the same notebook go step by step analyzing a given dataset in detail, and then separately designate what should be shown in the server context, which is very handy; the boss sees one view, I work on another, and it all stays in sync. Voila works with notebook cell outputs only, and so I don't think it's possible to have fully different views of your data in the two contexts. - Panel supports building complex GUIs, with hierarchies of nested objects that each define their own editable parameters, without having to tie any of that code to Jupyter, Bokeh, or any other GUI or plotting system. This approach is really important for building large, complex codebases (e.g. simulators or data-analysis systems) that sometimes are used in dashboards, sometimes in notebooks, sometimes in batch runs, and sometimes on e.g. large remote computing systems.

I'm sure there are lots more differences, but that's enough for now!

Panel can use a Bokeh server but does not require it; it is equally happy communicating over Bokeh Server's or Jupyter's communication channels. Panel doesn't currently support using ipywidgets, nor does Voila currently support Bokeh plots or widgets, but the maintainers of both Panel and Voila have recently worked out mechanisms for using Panel or Bokeh objects in ipywidgets or using ipywidgets in Panels, which should be ready soon. I'm not sure of the details of how one would use Voila with other languages, but Panel can already show anything that has an interface to Python, e.g. an R ggplot visualization.

I think this was already said above, but it still seems to be getting confused, so to repeat: Datashader renders everything out of core, in the server. So it doesn't matter whether a client could successfully accumulate results for a large data incrementally; to use WebGL directly one still has to send all of the data to the client eventually. With Datashader the dataset is never sent to the client in the first place; it stays on the server, which could be a remote HPC system with thousands of cores processing petabytes. Datashader renders the data into an image-shaped array on the server, then sends that (much smaller) array to the client, so that the client never sees any data larger than the available screen resolution. This is no claim that doing so is unprecedented or some crazy new idea, just that Datashader lets you render datasets regardless of their size, completely independently of any client (browser) limitations, and without having to serialize the data over an internet connection.

Sigh. Datashader is not a paper, it's an actual usable piece of software, so it should be compared to other tools and libraries for rendering data. Unlike nearly ever other 2D plotting library available for Python, it can operate in core or out of core, so it's entirely appropriate to advertise that fact (why hide it?). Unlike OpenGL's point drawing functions and nearly every other 2D plotting library available for Python, it avoids overplotting and z-ordering issues that make visualizations misleading (so why hide that?). Unlike NumPy's histogram2D, it allows you to define what it means to aggregate the contents of each bin (mean, min, std, etc.), to focus on different aspects of your data. It's a mystery to me why you think Datashader should somehow fail to advertise what it's useful for!

I'm not sure if you're objecting to the name "Datashader", but surely every library needs a name, and this one is accurate in that it allows the sort of shading that one does for 3D rendering to be applied to 2D data plotting. Or are there other buzzwords used in the docs you find objectionable?

By default, Datashader accurately conveys the shape of the distribution in a way that the human visual system can process. If you want a linear representation, you can do that easily; see the first plot in http://datashader.org/topics/census.html , but you'll quickly see that the resulting plot completely fails to show that there are any patterns anywhere besides the top few population hotspots, which is highly unrepresentative of the actual patterns in this data. There is no saturation here; what it's doing in the homepage image is basically a rank-order encoding, where the top brightness value is indeed shared by several high-population pixes, the next brightness value is shared by the next batch of populations, etc. Given only 256 possible values, there has to be some grouping, but it's not saturating.

Datashader is server-side rendering, and thus not in any way comparable to WebGL in its usage. With Datashader, only the final rendered/rasterized image-like object is sent to the client, which lets it handle arbitrarily large datasets (anything your remote servers can process). With WebGL the dataset is sent to the browser for rendering, which has some advantages but is a very different process than what Datashader does.

Geographic maps aren't the primary use for datashader; those are just easy examples that people can appreciate without a lot of explanation. In practice we use it for any large datasets that we don't want to subsample before visualizing them.

Yes, those are symmetric icons, a type of attractor. See http://datashader.org/topics/strange_attractors.html for all the details.

If you want one word, Datashader is a rasterizer. It takes data of many types (points, lines, grids, meshes) and creates a regular grid where each grid cell's value is a well defined function of the incoming data. Not sure anyone would be any happier with "rasterizer" than "renderer" or "shader" or any other single word...

Datashader's approach is a bit different from an accumulation buffer, though similar in principle. It's not 3D rendering, and has no need for a z ordering; instead it's essentially 2D histogramming. For points, it simply takes each point, calculates which pixel it would land in, and aggregates per pixel, without ever storing all the data points per pixel. The key benefit over something like SciPy's histogram2d functions are in how it is implemented and used -- highly optimized, and highly integrated with viz tools so that it can let you just interact with your data naturally as if you had infinite resolution. Try it and see!

Datashader itself renders networks: http://datashader.org/user_guide/7_Networks.html

The attractors at http://datashader.org/topics/strange_attractors.html are probably closer to art than science...

Of course! HoloViews does allow infinite customizability, to pull out more and more subtle features, show things more clearly, and just to make it look nice or to match your favorite style. But unlike ggplot2, HoloViews does that in a way that can apply to the _data_, rather than having to recapitulate the process every single time you build an individual plot. That way you and your colleagues can together build up whatever style you find most effective, then keep working with it across the full multidimensional landscape of data that you work with in a particular field. HoloViews is a completely different approach, if you really let the ideas sink in (e.g. from our paper about it at http://conference.scipy.org/proceedings/scipy2015/pdfs/jean-...), and is in no way a second-class citizen compared to ggplot2 or any other approach in R...

Personally, I don't _want_ a grammar of graphics; I want a grammar of data, where the data happens to have a graphical representation. I don't want to spend ages piecing together a fancy plot; I want to spend just a little time annotating my data to declare what it means, and then no matter how I slice and dice my data it will show up in a meaningful way. That way I can explore it to really understand it, which is the point of HoloViews (http://holoviews.org). But people approach plotting in lots of different ways, and some people actually _do_ want to spend their time making plots, so they are welcome to their ggplot2!

There are links to each library included, which will let you look at galleries of examples that are much more helpful than any single image would be. But if you want to collect images from each library, feel free to post that in the comments!

After 5 years and 40 releases, the Bokeh plotting library for Python has reached version 1.0 API stability.

And Python's duck typing is for addressing the needless constraints that statically typed polymorphism introduces. Discussions like this can go on forever! :-)

Sounds like we'll have to agree to disagree about a lot of things! :-) Having types specified statically is a recipe for generating untold many different versions of the same thing, plus untold bits of glue between them, which leads to huge codebases that need hundreds of people to work on them. Python was designed to avoid huge codebases and let smaller numbers of programmers be more efficient. (Of course, so was Haskell, but that's mostly a straw man, given how few large-scale users of Haskell there are in industry...)

After a decade of experience developing big systems in C and C++ followed by a decade of developing big systems in Python, my own experience is that static vs. dynamic typing isn't actually the issue that normally matters in a big system. Static typing is great for optimizing performance and for having efficient interaction with the underlying system libraries. So the closer your work is to the underlying machine and its types, the more important static typing is. But only a small part of any big system is usually like that, and static typing makes code harder to develop, harder to read, and needlessly limited in scope (overly specifying down to a machine type, when the actual algorithm may be far more general than that). Dynamic typing addresses those concerns, but it makes for a brittle system where each component makes a lot of undocumented assumptions about what might be passed in, with users of those components often just crossing their fingers about what the component might accept. Worse, dynamically typed systems eventually end up accruing pages and pages of hand-written documentation, assertions, and exceptions as people try to make them more well validated against user and external-programmer errors. All that well-intentioned "hardening" inevitably eliminates the readability benefit, except during initial development (when such checking is conveniently skipped). So neither static nor dynamic typing alone makes it easy to build big systems.

What does make it easy to build big systems is semantic typing. I.e., the ability not just to declare a machine type, but to declare the actual assumptions made by any component about its inputs. In some low-level code those assumptions are about machine types (argument i must be an 8-bit integer, and s must be a string). But in other cases the assumptions are about what the argument represents (argument s is a string denoting a filename (not just any string), argument n is a number between 1 and 36.2 (regardless of machine type)). Any given component makes some assumptions about what it can handle, and if you can easily declare those assumptions and validate against them, then the component is easy to read, easy to understand, easy to use, safe against user errors, and easy to implement (as you don't have to write any code to handle conditions not allowed by the declarations).

The beauty of Python is that semantic typing can be implemented very naturally within the language itself, which is crucial because it has to be extensible to handle any given domain's objects if it is to be of use. There are a variety of options, but my own package Param (https://ioam.github.io/param) provides support for semantic typing, and if you look at some of the examples there you should be able to get a feeling for how you can have the best of all worlds in Python: freedom where you can handle it, constraints where you want them, and everyone being up front about what they are doing in a way that makes big teams able to work freely with whatever other people come up with.