Python Data Visualization 2018: Why So Many Libraries?
anaconda.com
anaconda.com
I used it for charts in a paper recently, since it includes swarm plots. I hit a problem when overlaying certain types of plot on the same axes (I think it was swarm plots on top of box plots, so I could show every data point as well as the quartiles). The problem was that the data would end up shifted, so the x axis labels weren't correct, even though each plot on its own would work fine when the other was commented-out. The only reason I noticed this was because I spotted that a peak I knew occurred at x=13 was showing up at x=14.
Although it took a specific set of circumstances to trigger, I thought this was still a serious problem since it causes data to be misrepresented, and it doesn't cause any warning, etc. that something is wrong. I made a minimal example script, opened a github issue ( https://github.com/mwaskom/seaborn/issues/1409 ), where the library author insulted me and locked the issue. I sent them a followup email, to explain that I was not after "free tech support" (I'd already worked around the issue for my paper by 'faking' the labels), I wanted to help improve the library so that others would avoid having incorrect plots (especially those not lucky enough to spot it like I was) and how I'd already spent considerable time narrowing down the problem to the minimal example, as evidence that I wasn't trying to be a freeloader. I also began my email with an apology for using this side channel, and that I wouldn't contact them again unless they consented to it. The author replied with more insults.
I didn't contact them again, as promised, but now I'm actively opposed to anyone using this project, due to the author's complete disregard for corruption of scientific data.
Seaborn is a really nice way to take off matplotlib overhead and make your non-seaborn plots look better too. I highly recommend it.
It's every open source author's right to be a jerk and blow off people asking for help! Blaming and patronising the problem reporter is not necessary.
My problem wasn't "I need these particular graphs to look right", my problem was "Using this library in a seemingly reasonable way can silently produce corrupt/misleading results". That's a problem for all potential new users, and hence why I decided to invest a bunch of time in narrowing it down, report and pursue it.
I also didn't call the software itself objectively wrong; only the decisions/execution trace which lead to the incorrect graphs:
"this script doesn't set any axis labels or ticks: those decisions are made by matplotlib and/or seaborn, and are objectively wrong in this situation, regardless of the algorithmic details of how they have been arrived at."
I wouldn't have minded if it were closed as unfixable or something, but it was instead mischaracterised:
"This is, in my opinion, a legitimate issue worthy of acknowledging (even if such acknowledgement takes the form of a "won't fix" closure)."
I wouldn't recommend Seaborn to anyone who wasn't prepared to manually cross-check all of the entries and statistics in the resulting plots.
- It's great that I can manipulate a view by zooming, panning, turning a series off, whatever, but all that state is lost when I send the URL of the plot to someone else (eg, for the case where it was generated by a CI job or something rather than locally on my machine). It should be easier in Bokeh to grab all that state and stuff it into a querystring that can then rehydrate the same view later on.
- Doing plots with more than one y-axis is a lot more awkward and fiddly than it should be, and even once you succeed, the axes can't be panned independently.
- If your data series name overlaps with the label in your legend, the whole plot breaks in an extremely non-obvious way (basically, Bokeh thinks that you want a legend entry per row in the named data series). Even being fully prepared for it, I lose time to this every once in a while, and it's almost a rite of passage that every junior dev hits it and burns an hour or two trying to figure out what is wrong.
This is why my favorite Python visualization tools are not Python - I've been burned too many times by libraries coming and going, and I just don't have the time to spend farting around trying to track the latest library fads.
Also, for many of the visualizations I've worked with I'm fine to use an older library that's reached a stable point of development. It feels like the whole conversation about whether to build your new project on the latest web framework, or use something old and established like Django or Rails. Boring, relatively stable software in any area is a pleasure to work with if you really want to focus on your real-world problem more than you want to focus on newer tech.
That said, I understand it sucks to build on top of someone else's code only to have it be discontinued, but honestly it only takes a few minutes to judge a project's maturity. Here are some very easy rules of thumb. Don't write a lot of important code relying on a library that:
- Is younger than 3 years old. - Has less than 3-4 major contributors and 20 overall contributors. - Has lost steam: a lot of issues and pull requests open and stale with no triage tags, no discussion or responses. - Doesn't seem to have any automated testing / packaging infrastructure set up. - Doesn't seem to have a regular ongoing cycle of releases, be it long or short. - Doesn't have nicely laid out documentation. - Doesn't seem to have a user base, as indicated by a preponderance of questions and answers on stackoverflow, etc. - Just posted their "we made a cool new thing" post on HN a few weeks ago.
Yes, the cutoffs are arbitrary (and flexible!), but this hasn't failed me yet. The python world is filled with many wonderful and mature libraries. It just also has a lot of up and coming, promising young ones. Use whichever!
A refactoring and merging of these libraries to enable fewer people to maintain more functionality, such that the bus-factor of any given part of the functionality is higher.
In any case, the incentives are simply not there, as different projects have very different priorities (which is why there are so many projects). Some folks want to monetize their special sauce (Plotly). Some folks want to focus on high level statistical charting (Altair, Chartify) Some folks want to focus on interactive data exploration (Holoviews) Some folks want to focus on high performance and streaming (Bokeh) Some folks want to focus on high quality static image generation (MPL). The human and economical cost of getting all those groups under one tent is astronomical.
I see how you got confused, but I was making two separate assertions.
Assertion 1: If you merge the library projects together, and strip out the redundancies between them, then you'll have the same number of contributors, now distributed over fewer total lines of code. So each contributor can learn more of the codebase. (Bus factor goes up.)
Assertion 2: If you refactor the resulting libraries to reduce the total complexity (i.e. reduce the API surface from that of the union of all the merged-in libs), then you can begin to strip out technical debt from the project from the outside in. By eliminating now-dead code, you remove places where bugs can arise, and you lower the number of dependencies (which could otherwise have been sources of API-breaking changes when they update.) Thus, the number of people needed to maintain the project goes down. So the same total functionality can then be maintained by fewer contributors. (You don't actually remove contributors; they just become reserve capacity, with each contributor able to be less-overworked for the same result.)
Alas, it's not so simple. How much actual overlap is there to merge in Bokeh and Matplotlib, for instance? MPL renders images in Python and has no JS component. Bokeh does all of its rendering in JavaScript! The Python API is mostly just a thin wrapper around BokehJS. MPL has no server component at all. Neither of the does what Datashader does for large data sets. Neither of them has a high level statistical charting API, that's only in Seaborn or Chartify. Merging all these things together would cost a fortune in time and money, and at the end of the day not actually reduce the total codesize to any appreciable degree.
I often just skip the middlemen and use it directly.
Matplotlib isn't going away any time soon.
I’ve been using Matplotlib for the last 10 years and expect to continue using it for the next 20. The stability is there, you just need to resist the urge to keep switching. It’s like vi: whatever your “favourite” editor is, you know that vi will always be there.
Of course as others have said ggplot2 is better but I am going to check out plotnine now...
Then for other things people will choose based on more objective factors and therefore almost everyone will reach the same conclusion, so you will get only a few choices.
Also, people pointing out the diversity of needs behind the multiverse of plotting libraries in Python probably aren't aware of the large number of extensions built on top of ggplot2[1]. In python, they end up being completely new packages.
I think I would enjoy Python just as much if I figured out how to auto complete column names when writing code for my plots and pandas operations.
df.mycolumn
But what about df.sort_values(by='mycolumn')
In that kind of cases RStudios is smart enough to know that such an argument should be populated by a column name from the dataframe that function is being applied to.The point of having a graphical representation is to communicate with other humans therefore some subjective judgement is required; there’s no “one true plot” for each type of data.
Hadley Wickham's slightly abortive ggvis project was a wrapper around Vega.
Altair is the python library that wraps Vega Lite, making it even more concise to write chart specs.
One important unique feature is that Vega and Vega Lite (and Altair) provide declarative syntax for complex interactions such as cross filters.
I wrote a blog post recently with much more detail about all of this: https://medium.com/@robin.linacre/why-im-backing-vega-lite-a...
What should be easy is easy, what is hard is still possible - for me it's the perfect mix. The only (but a big one) drawdown is the peformance implication of the JSON generated for chart - over 50k lines of raw data for chart, you start to feel the lag.
I'll give it a shot! :)
You probably shouldn't be keeping data as pickles, for both compatibility and security reasons.
It's science research. My own simulation code and data. Some of it is hdf5, but pickle files are pretty convenient. Workflow is mainly turning data into plots. Heh, I thought I was doing alright since I'm not using textfiles.
It's 5-6 lines of code per chart for most examples here, with a few more lines for interaction (e.g. selection and linked brushing across graphs).
If I want pretty web-ish scatter plots Bokey or Plotly, and if I need desktop GUI visualizations, PyQtGraph, etc. Thankfully there’s no one saying we need a single toolkit for everything (though Matplotlib does try to do this to some extent)
It's fine you have multiple tools. The only thing that matters at the end of the day is the product you provide, so it doesn't seem to matter to me.
https://medium.com/@robin.linacre/why-im-backing-vega-lite-a...
For example, ROOT has the concept of ownership, entirely distinct from the python sense of references. A file owns the histograms that were read from it. Closing the file changes all references to histograms read from that file to None. It took a large amount of digging to figure out how that was even possible (overwriting an existing object from within the C API), but I cannot for the life of me understand why.
ROOT has way too many gotchas for me to ever recommend it.
I first noticed this effect with Python web-frameworks years ago. There was always some new framework that worked better or differently for some use case. (There are 73 web-frameworks listed in the python.org wiki.)
A few more examples from pypi.org searches:
“Reed-Solomon” 84 packages
“Elliptic-curve” 535 packages
“Nearest neighbor” 408 packages
“Simplex” (as in simplex method) 29
“Django” 10,000+
I just noticed a HN post about a RAFT implementation in Haskell. Pypi.org says there are 22 hits for RAFT in the python package index.
Also 3d plotting and graphing needs much more features. Trying to plot vectors in 3d, I found things are still very rudimentary. Even for surfaces, if you want to plot an ellipsoid, for eg, you need to reformulate the surface in polar form, then only matplotlib is able to generate it, were as Mathematica can generate surfaces from Cartesian expressions.
If this is true I would say it's a positive feature of python, but also a reason for fragmentation. You tend to see less fragmentation in earlier and lower level technologies. This is partially due to time, only good software or hardware lasts. But I also hypothesis it was because things are harder down there, you need bigger teams and more resources, which tends towards centralization.
There’s a good idea there, but D3 just isn’t quite right.
What eventually happened was that the BDFL blessed Django and most of the others withered. Some had enough ecosystem, or were components of some other larger project, or had a niche advantage of Django, and they managed to survive.
I think the reason why web frameworks and data viz systems proliferate in Python is just that they are so easy to write, yet still challenging enough to be really fun, and you get a lot of highly, uh, visible feedback and reward for doing it.
And that's not even counting differences of opinion on aesthetics...
Python has a multitude of vis systems because there are multiple use-cases, sometimes overlapping and sometimes not, and different people prioritize these very differently (and that's OK).
I should have said maybe, "relatively easy in Python".
Most of my students are used to Excel and are new to programming.
Expensive proprietary packages like Tableau(and even Power BI) provide a much better experience when it comes to presenting your results to management.
So we do the data gathering/wrangling/munching/analysis in Python but export the results for commercial visualization packages.
I can't in good conscience recommend any one Python library for visualization.
I mean for people who are just getting a hang of Python not Wes McKinney :)
Also, by the time you're leveraging SQL extensions, are you really still "using SQL"?
But if I said that, I have to be a pedant with no joy in their life, so...