How to learn D3.js
wattenberger.com
wattenberger.com
DC combines D3 and crossfilter for multidimensional filtering so you easily do things like click an element to use it as a filter for your other elements.
D3 is stunning. I've been looking for a nice visualization framework for some internal reports, and this (D3 or I guess DC) looks like a great tool to add some spice to boring charts.
Javascript frameworks are almost overwhelming in the numbers and functionality now widely available. If only there were more hours in a day to spend learning them.
You don't have to be constantly tweaking and tweaking everything all the time. Crossfilter is like the I-Beam of construction.
I wish we could do this with more computer science parts. Building something amazing and leave it alone. Move forward. Stop spending all our time inventing and learning new libraries, frameworks, architectures all the time and get stuff done!
You are totally correct by the way. I do agree that most software creators hand wave away the potential for their dependencies change faster than they can maintain their own product.
i->o is a common typo; here it could be negating the intended meaning!
It also appears that your demo for multi-chart is:
1. Broken
2. Blocked by most browsers and requires manual override of security suggestions (probably related to https and some sort of cross-domain things going on).
I'd be curious to see any working examples like the one dc.js has on it's main doc page. Cheers
I launched this approx 2 years ago - using it in prod, but not had proper time to devote on better doc/demo/presentation. Looking forward to it during current migration to lit-element.
I do wish that there was a way to handle NaN in crossfilter. The highly functional style codebase for dc.js also makes it pretty hard to wrap your mind around the codebase and easily extend it. Still, it is a really amazing library and I'm always amazed at how useful the software we've build on top of dc.js is for data exploration.
I've just been adding DC line charts and about to add scatterplots but was considering plotly.js instead. I'll be dealing with probably a couple thousand data points so performance may be critical?
My (mostly abandoned) blog has a bit more info on it, but isn't critical reading [2]
[1] - https://github.com/HamsterHuey/dcjs-canvas-scatterplot
[2] - https://www.intothevoid.io/data-visualization/dc-js-canvas-s...
This article gives a great intro to D3.js, and I'll be looking into it more. A colleague also recently showed me Dash by Plotly [0] which may be of interest to folks in a similar situation.
Not even Leaflet? I've been using its R interface and I think it's fairly easy to pick up.
I know. I spend much of my days writing 'high performance' Python, and Python is still my go to tool for just about everything. Julia's big advantage is, kind of like Fortran vs C back in the day, that it's fast even if you're just writing things in the most obvious and natural way for your domain.
There is no endless RAM, even for browsers :)
DOM-nodes are rather heavy compared to simple JS objects, so getting rid of them can bring some visualizations that are too heavy for the browser back into the realm of doable.
After that you have to optimize your other data-structures. If you have objects with many boolean fields or strings that are essentially used as enums, you can try to replace them with bit-fields, I read some library creators (BlueBird promise) got much performance gains with them.
https://wattenberger.com/blog/d3#drawing-svg-shapes
Canvas is way more performant than SVG for a lot of shapes, since they don't have the overhead of a DOM node.
But in general, you want to work outside of the browser if you have that much data. What I'll usually do is either:
- write a python script to process my data and use d3 for the final visualization, or
- have an API that returns a sliced portion of the data and create an interface that will request different sliced datasets when interacting with filters, controls, etc.
This election dashboard is a good example:
https://currents.parsely.com/election
it hits two endpoints, which return basically the maximum amount of data you can expect a browser to handle (the massive dataset is processed elsewhere). You could create an endpoint to return different views when you click on, eg, a different date, although then the user would have to wait for the data to load.
As a non-javascript user who does data visualization, I would love to have some advice about which technology(es) to invest time into
I used d3.js back in 2016 and it was already at that time very nice, but I always had problems to keep control of it (a lot of code and the end-results sometimes didn't work 100% as intended as my code did not cover all possible constellations of potential input data).
If I remember correctly after a while I then tried Cytoscape ( https://cytoscape.org ) and coding was a lot simpler and the results were more stable. I then dropped the project but I left with a positive impression of Cytoscape. There was maybe as well some other good candidate, but I just cannot remember anymore... (something that was doing at that time initial experiments with WebGL as optional/secondary output-engine - was it Cytoscape or something else?) .
Of course if you need max flexibility and very special results then D3 is #1.
https://jordi.platinum.edu.pl/xsplines/splines.html
Some day, I should try to work and get x-splines into more drawing programs; I really do find them a lot easier to work with than other kinds of splines.
I've done small frontends in Angular 1, pure JS, jQuery, and fiddled around with newer Angular and Vue. While I found Vue pretty intuitive, I kind of got stuck jumping from the little tutorial components to a full page - how to organzie components, the code modules themselves, etc.
It seems React is eating the frontend world, so I've always kind of kept it on my to-to list.
To give you a bit of context, the _large_ majority of people buy the advanced package (which really surprised me). The advanced package really just tries to bring all of your new knowledge together, going over d3 + React and d3 + Angular. It also has detailed walkthroughs of 3 complex data vizes, which could be really helpful. But both versions give you a great foundation to build off of.
* Use one of those higher-level libraries (bokeh, plotly) to create a chart that looks exactly like every other chart made by that library
* If you need to do anything custom, at all, you end up having to build from the ground up w d3.
In other words, those libraries are super cool and I use them on the reg, but they offer little to no customization, so people very often end up back at the bare-metal build-it-yourself-w-d3.
Here's the full Javascript reference for all of these attributes: https://plot.ly/javascript/reference/
How would you go about creating a live stock chart graph with D3.js?
All of the tutorials I have read assumed that the graph data was set in stone. Take this data, produce this graph.
I have created a few mathematical indicators for equity price predictions based on live data feeds, and I want to visualize them in real time in a graph.
Do you just keep rebuilding the graph every time a new datapoint comes in?
d3's power is that it is very customizable and it is surprisingly easy to just think of a chart format you need and implement it (and it is very easy to update as new data comes in, you can reload individual chart elements which is very cheap).
But...that comes at the cost of all the extra code needed to build the chart. If you are just building a plain-old chart, I would look elsewhere.
I have a data-heavy application. There is some stuff that is really domain knowledge and building those charts with d3 is worth it. But for the basic line charts or whatever, I just use plotly. Yes, that means a heavier page but it is worth it to move the complexity of multiple chart types out of my code.
My conclusion was to just draw SVGs on my own and skip d3. Well I used it for some of the nicer function like determining nice axis label spacing and formatting strings. But, I basiclaly cut d3 out of the loop and things are better off.
If you know what you want your graphics to look like and they are relatively basic (maybe d3 is better a complicated charts, I didn't get that deep) then skip d3 and just create the SVGs themselves.
https://bost.ocks.org/mike/join/
I personally usually am working in a React app, so I would use that to render (and update) the chart.
However, that's a pretty complicated example to start with! You might be better off using one of the higher-level libraries, like this one:
https://apexcharts.com/javascript-chart-demos/candlestick-ch...
There are a ton to choose from (just google `js candlestick chart`), and they can be pretty customizeable.
partition = data => {
const root = d3.hierarchy(data)
.sum(d => d.value)
.sort((a, b) => b.value - a.value);
return d3.partition()
.size([2 * Math.PI, root.height + 1])
(root);
}
* Zooming is non-trivial, you have to recreate it in React proper
* React has to re-render entire hierarchies or massive lists depending on the traversal method that you choose to use for hierarchical datasets (topological or depth first) and React chokes on these pretty easily since they tend to be wide and flat
* Quickly changing data sets can't really benefit from memoization or React.memo, since the layouts themselves will constantly change. It makes things a bit awkward when mixing with memoizable data.
However, once we got the groundwork down, we never really had to touch it again. It was much easier to maintain than the pure d3 implementation using useEffect or componentDidMount, since we didn't have to deal with any sort of DOM state ourselves any more.
https://github.com/kay-is/react-from-zero/blob/master/16-adv...
This is largely why I wanted to write a post like this - often "using d3.js" is what we think we need to do, but I wanted to show that most of the API is just utility functions. Here are the two modules devoted to manipulating the DOM:
https://wattenberger.com/blog/d3#manipulating-the-dom
I recommend (and follow) that you render elements like you normally would, and use things like d3 scales or d3.line() to convert your data into the physical aspects that are needed. I find it really hacky to do it any other way. I want to explore more, but I did a quick test to see which is more performant (letting React render a scatter plot, or letting d3 render a scatter plot), and pure React was 4x faster:
https://twitter.com/Wattenberger/status/1123413424678027265
Of course, this was just one example, but it agrees with my hypothesis.
I do have a whole chapter devoted to d3 + React (and one on d3 + Angular) that might be worth checking out. I think it largely comes down to understanding your tools and being able to pick the parts of d3 that will be most useful to you.
I essentially have a visualization class with an svg, and an update function for all the D3 stuff. The data is handed using props.
When the component mounts, or updates I call my own update function which sets the svg size, binds the data, and defines the enter/update/exit actions+transitions. Because I call update in the componentDidUpdate function, I had to catch 'fake' updates where the data hadn't changed, and abstain from state-updates such that I don't end up with infinite update loops.
componentDidMount() {
this.update();
}
componentDidUpdate(prevProps, prevState) {
this.update();
}
render() {
return (
<div ref={this.gridRef} className={"VisContainer"}>
<svg className={"VisSVG"} ref={this.svgRef} />
</div>
);
}
If you're curious, the first chapter is free to download and gives a good sense of the flow:
A better approach would have been to leave the old code running with stubs to the new method signatures. As it is, if you want to migrate code from V3 to V4, it takes a lot of mostly annoying and tedious work that could have been done once by a little bit of code that calls the new code from the old method signature. Have a d3V3toV4.js translator library in there if you want, but don't make people update years of code to get on the latest versions of the API.
I was litteraly unable to use the current version and went to implement v3 code based on tutorial and demo availability.
To this day I'm still planning to port everything to the current d3 version some time in the future.
all examples on Observable are up-to-date, so searching on here could be useful:
and if you find a bl.ock that you really want to port, this changelog could be helpful:
https://github.com/d3/d3/blob/master/CHANGES.md#changes-in-d...
most of the changes are simple name changes that put the methods directly on `d3`: eg. from `d3.scale.linear()` to `d3.scaleLinear()`
The other big problem with d3 is that you don't get much for free with it. Many people are surprised that you have to draw a bar chart using rectangles. But d3 doesn't "draw graphs", it just maps values from your data into attributes in the DOM. You have to know HTML, SVG, CSS, and, of course, Javascript very well already to have any success with d3.
`.enter()` and `.exit()` is not covered at all, simply outsourced to the guide/demo by Mike Bostock, which is actually a really nice demo however it also omits explaining the mechanism.
I also specifically didn't want to go into the data join patterns for two main reasons:
1. there are lots of other articles that talk about it in depth, and 2. these days, I largely don't use it at all.
This is partially why I wanted to write this up, because the DOM manipulation parts can seem like an integral part of D3, but there are really only 2 modules devoted to it. I mostly use D3 for its utility methods, and a JS framework like React to handle all of the rendering, and I really want other people to be able to discover those functions without first "learning d3".
These days I also tend not to use d3 for DOM manipulation, but only for its powerful helper functions (d3-scale, d3-array, d3-shape, d3-geo etc.)
I even try not to import d3-selection except for very specific cases.
I’m using Vue for DOM manipulation, and computed properties play very well with d3 helper functions (i.e. a scale which auto-updates itself reactively). In a way it’s very similar to ObservableHQ’s reactivity. Most examples from Observable can be relatively easily converted to Vue with computed properties.
I've had a couple projects where I wanted to dive into D3.j but I found it to be a huge topic and most of the time I end up with a D3 related framework where I don't have to dive in to the depths of D3.... but still wishing I could learn D3.js directly.
No DS records found for [retracted].com in the com zone
No DNSKEY records found
[retracted].com A RR has value [retracted]
No RRSIGs found
https://vega.github.io/vega-lite/
They have a python graphing lib too based on the same “grammar of graphics”
The interactive crossfilter part in particular
https://altair-viz.github.io/user_guide/interactions.html
https://vega.github.io/vega-lite/examples/interactive_seattl...
D3.js hasn't changed radically in the past ~8 years, mostly because it doesn't need to. The SVG API has been fairly stable (at least as far as what has been implemented across browsers), and the basics of visualizing data are static (eg. how to convert data values to physical values using scales.
Once you know those two things, I consider it unmatched for creating unique visualizations, particularly interactive ones.
[0]: https://towardsdatascience.com/collect-your-own-fitbit-data-...