Scientific Visualization: Python + Matplotlib (2021) | Hacker News Reader

Scientific Visualization: Python + Matplotlib (2021)

github.com

177 pointsdanso54 comments

I work with chemical and materials companies on full stack data capabilities and interactive viz / dashboarding is a recurring pain point. Frequently there are larger scale processes (beyond a single engineer’s or scientist’s scope) that still require scientific-level interactive viz that the likes of PowerBI or Tableau can’t (or won’t?) provide… if your company even has a subscription. Things like being able to dynamically re-group/nest variables and recalculate statistical tests, dual axis capabilities, automatic sig fig number reformatting, just all kinds of quality-of-life features that in some cases are possible but in most cases are considered extreme edge cases and require too much manual config / aren’t templatizable.

Of course on the other end you’ve got the whole Python/matplotlib/seaborn/bokeh/plotly/vega/altair “ecosystem” (although it’s more of a swamp if you ask me), which require someone to maintain Python code and a means to stand up an internal server. Not to mention that most use cases require significant customization. Plotly Dash always seems somewhat promising but as someone below mentioned it’s actually kind of slow? Every time I try it I’m just kind of underwhelmed.

I hear ggplot in R is good but I’ve never used R and it’s hard to get a critical mass of people in a company behind R so that’s kind of off the table.

The only programs that really get the aesthetics of scientific plotting right without a ton of customization are JMP, Origin, and Igor Pro (props if you’ve heard of it), but these are all desktop apps… although JMP is starting to make a push into cloud-hosted stuff.

I guess all that is to say if anyone is interested in starting a company in this space, let me know.

I’m working in biotech. I think creating “curated” materialized views with dbt + datawarehouse or duck db + using webgl with d3 will be best long term solution https://blog.scottlogic.com/2020/05/01/rendering-one-million... .

Matplotlib works well for static plots. Altair and others freeze at around 4000 data points, which is crazy. Streamlit + matplotlib is impossible to maintain but is quick to get up and running.

Exactly. This is the exact stack I envision to be the future in that space. The work of Scott Logic is awesome (and the company looks really nice as well! People and values)

I completely agree with the entire stack. I’ve basically been learning d3 for this exact reason - the primitives are so intuitive and I can tell I’ll be able to make what I want. And yeah streamlit is so close to being useful but just not quite there. But isn’t plotly built on d3?

I agree with cactusfrog that d3 is a step in the right direction for dataviz. It really is a swamp out there, with twenty or so different ways to do similar things but usually not quite what you want (or at least not as easy as you want it to be)! I've been researching dataviz out of curiosity and annoyance with the current state of dataviz software for a while before this post popped up and would be interested in integrating a lot of the past decade-or-so of open-source work in this space to a tool for full stack data viz. I'd love to work with as many engineers who do data viz as I can! You can get in touch with me via email at puterproblems [at] proton.me if you're interested (and to find out more about me as I'm aware this is a brand new account :D).

Doesn't R require coding too? And my recollection of Igor Pro was from long ago (on a 68k Mac), but it also required coding in its bespoke scripting language. In fact I walked away from it for that exact reason... I wasn't going to spend brain cells on somebody's proprietary language.

I think that for things like dashboards, we're still stuck between "code" and "no code" tools. I don't know of a happy medium.

Vega/Altair is a declarative grammar, so I wouldn't dump it in with the others. It's also convenient because it reduces to json (easy to store) and has TypeScript libraries for native presentation in a browser.

Igor Pro is nice but the underlying language which grew out of a set of macros is pretty rough to work with (by default works via side effects and mutation of global state, though there are ways to contain them). However, I know people who have built some nice GUIs on top of it. While it does have HDF5 I/O integration, there's no memory mapping and it will choke on larger data sets from what I recall.

I've heard promising things about Makie [1] in Julia; there is also capability to build a dashboard called Genie [2] (and a commercial dashboard builder [3]) though not sure if Makie and Genie play nicely together at the moment.

[1] https://docs.makie.org/ [2] https://genieframework.com/ [3] https://info.juliahub.com/blog/create-low-code-apps-on-julia...

I think https://github.com/uwdata/mosaic is really promising here. See the example https://idl.uw.edu/mosaic/examples/linear-regression.html where the user can recalculate a linear regression based on their selection.

You'd still need to implement any custom selection widgets, data transformations (like other statistical tests) etc. still missing, but i like the technical design to build on top off. It uses https://github.com/observablehq/plot under the hood, which aims to have just as flexible a grammar as ggplot (already quite capable) but with interactive features (built by the creator of d3 and uses it under its hood).

For the folks who don't like the default style of matplotlib, fonts, colors etc... I made a stylesheet for my plots based on tailwindcss which you can use: https://www.bruderer.ai/blog/matplotlib

Additionally, Seaborn (https://seaborn.pydata.org/) is a great mention for people that want to use Matplotlib with better default aesthetics, amongst other conveniences:

"Seaborn is a Python data visualization library based on matplotlib. It provides a high-level interface for drawing attractive and informative statistical graphics."

Better is subjective. Not everyone likes Seaborn's defaults.

I really like the book's subchapter on colours, wish it was even more elaborated on. Colours are one of the subtle things I so often find difficult to get right.

As to using matplotlib in published research: when I started out as an undergrad, everybody in the research team used OriginLab for plotting -- my impression of it then was pretty good. At some point, I started using matplotlib + Latex + science plots and it caught on, mostly because there's no need to shift all the data around to a separate programme. Scienceplots package does heavy lifting with fonts and styling for specific journals, so it's just a matter of designing the right plot geometry and information density [1].

[1] an obligatory Tufte citation.

I can recommend proplot (https://github.com/proplot-dev/proplot) as a ”beautifier” wrapper for Matplotlib—particularly useful for scientific publications

SciencePlots is another option: https://github.com/garrettj403/SciencePlots

My weapon of choice is still R + dyplr + ggplot2 - mostly because I have been using it for so long and know it by heart.

I'd love to try something new, but don't feel the whole Python world is it. Is there any modern take - doesn't have to be production ready but should show a promising future? Anything from a more modern ecosystem, like Rust or zig, maybe?

It probably depends on your interests but I'm excited about the efforts using GPU rendering for interactivity (it's related to vispy but I forget the latest status of whether vispy is the current focus... I think the last I remember is someone was prototyping a vulkan-based system).

Edit: the vulkan-based system I was thinking of is datovis

Why on earth would you want to do exploratory data analysis in a compiled language

Not necessarily in Rust itself. Actually I like R as a language. I was more thinking something along the lines of Typist. A modern take of a proven concept, possibly with a Rust inspired scripting language.

https://typst.app/docs/reference/scripting

Is ggplot2 still using hyphens for minus signs? A pet peeve of mine.

plotnine is a really good grammar of graphics at this point.

Love Mr. Rougier's open books. I supported this one while it was in development and I wasn't disappointment by the final product.

I've long awaited for him to finish "Python & OpenGL for Scientific Visualization" [0] but I'll take this in the mean time :P

[0] https://www.labri.fr/perso/nrougier/python-opengl/#python-op...

Very nice guy too! Met him at a workshop recently and got a hardcopy of the book

I second this! Nicolas and I have been working hard lately to try to advance the field of scientific visualization in Python — stay tuned!

Looks good, thanks

As an Engineer (but in a scientific field) I just love Matplotlib. I moved a number of years ago from Matlab and even Excel (love how I still see papers with that mid grey background and full magenta on figures).

I'm a poor data analyst and programmer but have always managed to do what I need, mostly just time series and scatter plots. Sometimes a little more involved for my ability: such as a set of heat maps with common scaling and some tiles omitted...

I see some comments about the quality of the rendering and the look, but I think the default is really good. Nice proportionality of text, lineweight etc IMO.

Anyway I'll get to my point. I really can't get any value out of the Matplotlib official documentation. I'm not going to criticize it, just say that its not compatible with my brain. On this basis, good quality and accessible literature like this is very well received.

I find the same thing about the official docs. I can't really put my finger on the reason, except that it might be due to the vastness of MPL and the need for the docs to auto-update with each revision cycle.

When I do find useful docs, they're usually in the form of an example that I can use as a starting point. Referring to the docs comes later if needed. The consequence is that I probably use only 1% of MPL's capabilities, but that's already more than I could ever have imagined.

I've only been using Google Copilot for a few weeks, so it's too soon to know if that's how I'll deal with the situation in the longer term.

Some of these examples are surprisingly decent. When I tried matplotlib I found it was ugly by default; I guess it has improved a lot.

For publication quality graphs I tend to use GLE. Not sure why it isn't better known, but check out the examples here:

https://glx.sourceforge.io/examples/2dplots/index.html

Maybe I can’t see the forest for the trees but is there an html version available?

Python ^ ( matplotlib v pyqtgraph ) is my goto

Nice book. But frankly Tikz and pgfplots are better for scientific visualization, unless your plot is too complex (in which case you have to settle for a lower quality plot).

How do you use those outside of Latex? Lots of figures are coming from Matlab or Python, where Latex is a non-starter. Is there a cookbook to pollinate?

You export figure in PDF or some other format.

TikZ has a steep learning curve and felt so convoluted when I tried to use it.

Great use case for llms?

I would actually be very surprised if that is true. tikz is one of those examples (like e.g. M4 macros or assembly code or OpenSCAD) where producing working code requires significant mental arithmetic. Most skilled humans need to write down intermediate calculations on a piece of paper.

If you don't believe me, look at the "tikz unicorn" of GPT4 demo fame. Then try to replicate that with the currently available "degraded because of safety" GPT4 versions. Then ask yourself how on Earth you could use that to produce publication quality graphics.

I tried the "Draw a unicorn in TiKZ" in chatGPT now, and it actually looks even better than in that paper, and the code is very clean too. Not surprising as the quality of outputs from chatbots have steadily improved in the ~17 months since.

TikZ does have a steep learning curve and can get pretty convoluted, but once you get a hang of it, you have a tremendous amount of control over what you are doing. I sometimes feel like TikZ is more like TeX instead of LaTeX given how primitive it can be, but it can be quite powerful if you give it enough time.

Matplotlib is very easy to customize via “style sheets”, so you can make it match any style you want. Note that you can even turn on the “usetex” option and provide a preamble to compile all text/equations via TeX.

No doubt it’s very customizable. But the display quality seems to me not as good as even matlab. At least with defaults, which is a bit barebones. You have to work on it .

The syntax of python is also a bit verbose. Like, why do we need plt.show(), when I plot a figure obviously I want to plot it! Now I know, the point is, see matlab, gnuplot etc.

Compare with the default plot of gnuplot.

The limitation of latex is you can’t do much calculation with data, and memory usage.

Don’t get me wrong, I still sometimes go to python for plotting. But it’s general purpose programming language, and for specific applications there are custom tools.

Don’t get me wrong either – I personally prefer Gnuplot by quite a margin, Matplotlib is indeed too verbose. These days I use Matplotlib more mainly because I use Python more.

I’m just saying that it doesn’t take too much configuration to make publication-quality plots in Matplotlib either, by e.g. enabling the TeX integration and setting up a reusable stylesheet file. The style sheet format is actually very reasonable, and several examples ship with Matplotlib. There’s also the Seaborn package if you want some nicer default stylesheets.

(Side note, the same goes for Gnuplot… As shown by e.g. gnuplotting.org you can definitely create publication-quality plots there as well.)

gnuplot + tikz terminal. I have yet to see better looking figures.

And technically, it's one of the rare if not the only one solution that can give you bibtex resolved citations in figure labels.... (Don't do it, the journals hate it)

> The syntax of python is also a bit verbose. Like, why do we need plt.show(), when I plot a figure obviously I want to plot it!

That is kind of annoying. Fortunately, it's easy to turn that off by activating "interactive mode", in which figures automatically appear when you use a plotting command. You can do this via "plt.ion()", or have it set automatically with

   interactive : True

in your .matplotlibrc file.

Just keep in mind that when you do this, most plots will by default reuse the existing frame without clearing it. So you need to manually call plt.figure() more often (which in my opinion is more logical than plt.show).

There’s also a change in semantics regarding whether the script “hangs” until the user closes the plot window, if I recall correctly.

Or you can call plt.clf() to clear the frame.

I often use Matplotlib within a GUI. It's convenient to be able to defer the display of a plot until it's fully built, for instance if the plot is displaying data in a live update mode.

That's the whole point. It's not at all obvious that you want to show the plot.

It's just as valid to save the plot and use the file in an external document.

You can use matplotlib's pgf backend for direct use in latex. That said, not sure I agree that's better for scientific viz. Perhaps in the past. Also tikz/pgf is much more fiddly and has it's limitations too (imho).

Please for the love of reproducibly, interactivity, and not having 200 system dependencies use plotly instead of matplotlib.

Interactivity is not desirable in all situations. For static figures, matplotlib is still king.

Can plotly generate publication ready vector graphics with Latex texts?

plotly has had ten years to take over the world?

Plotly is only useful if you need to display low hundreds of data points. For anything more, it becomes so slow it's quite literally unusable.

Reply on news.ycombinator.com