The Trouble with D3
medium.com
medium.com
Edit: well, browsing zrender's source, I see svg logic too
If you need standard stuff, there are plenty of awesome plotting libraries that utilize D3 on the back-end while abstracting the complexity away from you. Some even try to give you some amount of flexibility and composability to customize them, but they are always going to have limitations compared to the raw power and flexibility you have if you roll your own D3 visualization.
You can't have it both ways, and that's something a lot of people don't seem to understand. If you're absolutely sure that a visualization library can support all your requirements, then it is absolutely the easier way to go about doing things. But you just have to be really sure that your requirements aren't going to change past the limitations of the visualization library of choice.
In data visualization, creating a scale based on the domain of your data (the range of your data) is annoying. This is handled for you by utility libraries in d3. It also can draw the line ticks of the scale without issues.
Also placing labels is not too easy. D3 does that nicely.
The truth is you can do all of that yourself, but if I were to try it, I would go over the d3 utilities first just to see how complicated some of the visual aspects you haven’t thought of but will need are.
The drawing part of d3 (SVG, canvas) are not so hard to do on your own, but the way d3 isolates the context of what you are coding is very helpful. I’ve only run through the examples and read the d3 book as well as several other charting books, but I don’t use d3 on a day to day.
But let’s go over a more realistic example. In one of my books, the author writes about simple line charts and displaying the quartiles (made by standard deviations) of the data. None of my charting libraries handle that. Rolling my own would be very time consuming. Doing this in d3 would be more feasible than the alernatives.
D3 is awe-inspiring in its scope, and learning to use it has a steep learning curve. But for anything beyond trivial use cases the freedom it affords you is worth the effort.
Consider how much JS you'd have to write to even start creating all the frameworks to build in native SVG structures... a lot.
I created html graphs with tables in the late 90s, made Flash graphs in 2000s, upgrades to numerous JS based charting libraries after this, building custom graphs the whole way.
I had looked at D3 so many times over the years when it first showed up online. But it "looked" complicated, no obvious "chart" library, etc...
Finally, when I tried D3, I was like "why the f didn't I start with this?"
Basically, D3.js strikes a good balance between ease of use and expressive power, that leans pretty far towards expressive power, while still being far, far less work than rolling your own.
Somewhere else someone described D3 as a hammer, and that's a good analogy, because if you don't start with a hammer, it's the first tool you'll make for yourself when you need to pound nails.
In another case I might not care about it, but use d3-selection, a mechanism for creating a data-driven DOM tree — but it's pretty clear that you can achieve similar mechanics with React.
Importantly, you don't have to pull the whole thing into your dependencies just because you use a bit of it, and it's light on own dependencies. So just using a small subset of functionality is "cheap."
Such as: https://github.com/d3/d3-shape
https://github.com/d3/d3-scale
Doing it from scratch without the help of utilities like these would be much harder.
A couple examples:
var line = d3.line() .x(function(d) { return x(d.date); }) .y(function(d) { return y(d.value); }) .curve(d3.curveCatmullRom.alpha(0.5));
line(data); // gives you svg path
and
var color = d3.scaleLinear().domain([10, 100]).range(["brown", "steelblue"]);
color(20); // "#9a3439" color(50); // "#7b5167"
All of the functionality in d3 is split up into smaller modules that can be required independently.
It gives you a whole lot of helper functions that really speed things up, data binding, animations, transitions, etc.
For the vast majority of data visualizations that are of the standard type, d3 is a not a scalable way to do it. The lack of high level abstractions means developers can't be as productive can be, and the code base implemented for one svg visualization is likely not usable for another, unless you put abstraction wrappers around it, which is basically nvd3, or other d3 wrapper libraries..
If you want reusability, build your abstractions or use a charting library, but recognize that this will in-turn limit the flexibility of your reusable code.
You can't have your cake and eat it too. It's the very reason why D3 is still so relevant today.
Pros and Cons with both approaches no doubt.
After learning how it works, and more importantly why, it's more obvious why it's useful. It's a chart building toolkit, not a chart builder, and you can dream up all sorts of interesting things with it.
The upside is that the author has a huge library of examples you can build from [0]. It's a great tool!
https://developers.google.com/chart/
High quality, very good browser support, good docs, and well-maintained. There are wrappers out there for React/Vue/whatever too
...
svg.insert("g", "g")
.attr("fill", "none")
.attr("stroke", "steelblue")
.attr("stroke-linejoin", "round")
.selectAll("path")
...You might assume that g, fill, or stroke might be d3 terminology but it's really SVG! A quick Google of "svg g" will get you to https://developer.mozilla.org/en-US/docs/Web/SVG/Element/g
It blew my mind open, hope to share the same.
.chartline { stroke-width: 3px; color: red; }
D3 has some convenience functions in JS for creating SVG, but you don't have to use them, you could write SVG just like HTML, it will look more like this in the DOM: <g fill="none" stroke="steelblue" stroke-linejoin="round">
...
</g>It still takes me a crazy amount of time to get a simple chart or graph working in D3 if it's anything more than copy/pasting an example. The last thing I did was an area graph that you could click and drag one of the edges and it would update the numbers in the legend as you did it. It took me 20-30 hours to get it working, and I'm sure I re-invented multiple things that D3 provides because it was easier to just write them from scratch than trying to find where Bostock hid it in the library and then decoding whatever crazy abstractions he came up with for it. I think he's bad at naming stuff and also bad at writing docs.
By the time I get something working in D3, I've come close to the point where I could have just written all the abstractions myself. I can use a lightweight virtual DOM library and just write functions that take data and output DOM objects.
D3 does barely more than that, at the cost of learning some really funky abstractions.
I really wish there was a library out there like D3 that tried harder not to surprise me with every abstraction.
But, I think the problem is just a hard one, because I'm not aware of anything that's much better :(
But if you're doing anything remotely bespoke, with any sort of custom interaction or transition, or anything in which you need to multiplex several datasets in a single set of visualizations, this toolkit is hard to beat.
My favorite personal D3 use case so far is modified animated scatter plots/beeswarms using force layouts. Immortalized for me in the "Four Ways to Slice Obama’s 2013 Budget Proposal" ( https://archive.nytimes.com/www.nytimes.com/interactive/2012... ), I've found that set of techniques useful for visualizing business processes in my own work over the years.
I've yet to encounter anything else that comes close to enabling those kinds of data visualizations. It's the "funky abstractions" some folks have trouble with that enable me to swap out X or Y scales with intuitive animations by literally replacing a single object and then re-invoking an idempotent render() function. Working with D3 is the most productive I've ever been in my life from a data visualization standpoint, and I suspect that will continue to be the case for some years.
> I really wish there was a library out there like D3 that tried harder not to surprise me with every abstraction.
Interestingly enough, I find that the combination of React + inline SVG fill this space pretty nicely. The engine is well thought out and documented, as is SVG. And animations are easier to do with CSS anyway.
EDIT: note that React is declarative too, which helps with visualizations a lot.
Wrong tool for the job.
D3 fills the void between "just send my data to a bar chart" and "I need some charts, maybe I'll use raw JS to manipulate SVG's". You're probably in the former camp.
`The last thing I did was an area graph that you could click and drag one of the edges and it would update the numbers in the legend as you did it.`
That doesn't sound like "just send my data to a bar chart" to me.
The examples do "cubic" instead of just "connecting the dots" but luckily it can be turned off:
http://www.chartjs.org/samples/latest/charts/line/interpolat...
Running with the analogy in the article -- it is fruitless for someone who's never done any woodworking or furniture design ever to hope to become the next Charles Eames within the span of a few months.
Ive done a bunch of javascript, DOM and visualisations in OpenGL, so i feel i should have a good chance.
That said, I mostly learned it through the examples rather than a tutorial.
Depending on their background (as in the article) and specially their objectives I would recommend quite different things. HTML+CSS is the basic for ~80% of my recommendations so far, but some times I've recommended straight Javascript, python or even C++ (Arduino).
Heck, a couple of times I've recommended some people NOT to learn programming. Wanna start a business by next week? Test ideas with pre-made platforms and CMS. Read about startups, product-market fit, etc. If you are starting a business expect not to have time for learning programming from scratch.
I wish I saw that last bit explained in the article and I feel it's a big omission. Some times NOT learning a tool/paradigm _is_ the right choice. If you just want to show a simple graph by tomorrow, don't learn D3.js, use any of the existing charting libraries.
The result of things like this is that d3 encourages mindless copy/paste, and you either have no idea how it really works, or else you've scoured the d3 source to understand what's going on (which then requires expert level of javascript knowledge).
I spent the last week trying to understand the brush and zoom APIs, and everyone from stack overflow to medium to bl.ocks just seems to push a style of learning that lends itself to "just look at this example, and figure it out from there."
So while sort blog articles are not the best way to learn D3, I think a better job could be done in illustrating the mechanics, models and use cases through geometry, and explaining what subset of a particular API are absolutely required to achieve a goal.
Comparatively speaking, learning ggplot2 is a sublime experience. After about a week, you feel like you have super powers.
D3 does feel overcomplicated for just building static visualizations, but if you're building custom interactive stuff, learning it can be nearly as joyous an experience.
Whereas seaborn has similar pre-canned plots, but it's all built on matplotlib so you still have that flexibility if you want it.
If you can grasp this idea of visualisation as creating a mapping function, then D3 becomes a beautiful toolbox for assisting you. You decide which dimension of your data you want to represent as the horizontal dimension, which as the vertical, as color, as size, etc. You decide how you want to scale them. Then you use D3's high-order functions to create the appropriate individual mapping functions, compose them into your complete visualisation function, and pass it your domain data.
If you think of D3 as a "charting library", or a collection of visualisation classes or objects, then you're going to struggle. But the problem isn't with D3, it's that you haven't really understood the paradigm of the library.
Hmm. I built the first version of my quantum wavefunction visualizer with D3, since D3 has a reputation for being good at visualization. However I eventually dropped D3 for WebGL-backed three.js.
One problem is much of the SVG spec is just not implemented. This means it is less capable than it appears at first. 3D in particular was very awkward.
Performance was also a problem. D3 round trips through text, making it hard to achieve fast custom animations. A big performance win was to truncate floats to two decimal places, which is an absurd thing to need to do.
three.js had its own issues, in particular drawing lines which is quite challenging. However it was more flexible and faster.
http://ridiculousfish.com/wavefiz/ if anyone wants to see.
And more generally, http://www.chebfun.org
e.g. http://www.chebfun.org/examples/ode-eig/DoubleWell.html http://www.chebfun.org/examples/ode-eig/Eigenstates.html
Isn’t this just an inherent problem with using SVG (or DOM in general)?
Conceivably you could use D3 to drive some other drawing method, though you might not get that much out of it at that point, unless you were trying to draw maps or something.
But in any event SVG path strings (and other element attributes) are inherently strings, there’s nothing you can do to get around that.
GPUs are special-purpose hardware designed explicitly for tasks like rendering graphics, and WebGL is based on a version from the late 2000s, based on insights from generations of high-performance computer games.
SVG is an XML-based bureaucratic-committee-specified document format from the late 1990s with a bunch of weirdly designed graphical elements thrown in, largely based on a drawing model invented in the 1970s.
SVG2 is/was an initiative to modernize it (and is to be finalized these days based on actual implemented features) but browser vendors have indicated they're only implementing a small set of conservative features [1] considering many vector effect features for UI are already part of CSS.
When SVG2 is finalized, it will probably just consist of a description of those SVG 1.1 and 1.2 feature actually implemented in browsers.
[1]: https://nikosandronikos.github.io/svg2-info/svg2-feature-sup...
But it doesn’t fundamentally change the concept, which as I said is to be a 1990s-style XML-based document format with a bunch of (frankly inconsistent and arbitrary) graphical elements in it, based on a 1970s drawing model. YMMV.
It’s great that it exists because browsers don’t give web authors any alternative method to match many of its capabilities. But it’s just as much of a confusing and frustrating mess as all the other web specs.
The browser doesn’t have any better way to do it. The performance will likely be good enough for whatever they are trying to do; CPUs are super fast these days, and can render many thousands of graphical elements at 60 frames per second even if you are forced to render everything in a cache ignorant and super-high-overhead roundabout fashion, and could conceivably do it all 20x more efficiently on bare metal on the CPU, or 1000x faster on a GPU.
If performance turns out to be unusably slow, they’re basically screwed, because the alternative (writing some WebGL thing from scratch) is way way harder.
There are a bunch of inherent graphical problems baked into SVG implementations (like compositing done in the wrong color space, lack of support for reasonable color spaces, inaccurate rendering of very fine features, phantom rendering of hairline gaps between immediately adjacent shapes, poor text layout, many components which are too inflexible to handle straight-forward goals, weird inconsistencies with other common tools, various bugs and cross-implementation incompatibilities, etc. etc.) but implementing a whole vector graphics pipeline from scratch to fix those would take years of work from a team of experts, something akin to implementing a video game rendering engine.
Someone just trying to make diagrams should live within the constraints of SVG and try to be happy with what it is, maybe with occasional grumping on a web forum.
Most of the comments here complaining about D3 boil down to the following:
- Using the wrong tool for the job... if all you want is off-the-shelf visualizations, use one of the many visualization libraries out there that utilize d3.js on the back-end
- Not wanting to put in the effort to learn the basics of SVG, CSS and the D3 API, and yet expecting hand-holding and everything to just be easy
The beauty of D3 is how powerful it is and how unopinionated it is. You can literally make any type of visualization with it, and it has some astoundingly useful helper libraries for data binding, data manipulation, color handling, etc.
As someone who picked up D3.js from scratch, I learned it by having clear goals regarding what I wanted to achieve. Having built charts / visualizations that are used in several products, it became very clear, very early on, that unless your customer/product requirements are super light-weight and run-of-the-mill visualizations are okay, you are bound to get to a point sooner or later where a visualization library isn't going to cut it. At that point, you're either going to spend a ton of time trying to shoe-horn a feature into an existing library that isn't made to be flexible, or you could have just saved yourself a whole bunch of effort by having gone the route of writing your visualizations in D3 from the get-go.
D3 absolutely has a place in the data visualization space. You just need to be aware that it may not be the right tool for you, and if you do need to wield it, you need to put in the effort to actually familiarize yourself with the D3 related landscape rather than expecting to be able to simply copy-paste examples and hack your way to a working solution.
PS - I should mention, that I'm an ML Engineer / Data Scientist, not a data viz geek, so it's not like D3 is relegated to people who do data viz for a living. There are plenty of folks who have picked up D3 on the side and done amazing things with it.
Sadly, this problem has become endemic within the web development community in recent years. People are more concerned with learning the latest buzzword framework than the fundamentals.
The article itself quotes this tweet early on:
The trouble with D3 is to build a visualization you must also have a deep understanding of SVG, DOM, JavaScript, geometry, color spaces, data structures, the standard model, and quantum physics — @seecmb
Well, yes. Designing and implementing good visualisations is a skill. It has many aspects, and they aren't trivial. If you want to draw a standard, no-frills, minimally-customised chart then there are many higher-level libraries that will draw one for you and you're not the target audience for D3. If you want to have more customised and creative visualisations and interactions, you have to customise and create. That requires understanding the tools available and knowing what you're trying to achieve with them, but if you are willing to put the work in to reach that point, D3 offers a useful and powerful toolbox.
If anyone is looking for a React-based charts library check out Recharts at https://github.com/recharts/recharts.
It's fairly simple and in some cases, gives you the ability to overide functionality with your own functions/React components which is fairly sweet. So for instance, I have customised my tooltip.
Feedback like yours can be so valuable, especially with the maintainer of Victory here. There's just no way to replace the knowledge/lessons learned from real life developer user experience.
I picked up D3 and after a few days got to the "all I wanted was some graphs, not a graph framework factory" point. Vega is great for this.
If you look at a force-directed graph in D3, for example, you see how little the library gives you. It's up to you to decompose your data into nodes and edges, it's up to you to draw every shape, it's up to you to trigger a redraw using window.setInterval. And the examples are written in a dense functional, mutable style that uses all the advanced features of JS, SVG, and the DOM - and requires at least some understanding of academic visualisation jargon to boot.
I believe that Bostock took the W3C seriously and wrote a toolkit that "goes with the grain of the web", preferring not to reinvent (or abstract) anything that was already widely available. It exposes the simple truth that the surface area of a modern, standards-compliant browser is enormous, and extraordinarily demanding on the mind of the programmer. And also, the dynamic, functional nature of JavaScript points programmers toward a style that is very much like that espoused by D3.
And yet, I admire and use D3. In part, I think it's honest. If you program browsers for a living, you should master your dependencies! D3 really isn't there to hide the web from you, it entices you to learn the web that's already there. SVG is incredibly powerful and still underused by modern applications, IMHO. And no other library has the breadth and depth of layouts that D3 does, to my knowledge, and the layouts are the hard thing to program (especially performant layouts). So basically the price of admission is learning browsers really well (which takes years), and then you get access to all this sweet layout code for free! It's a good deal, if you ask me.
Additionally, you can animate those props using the Web Animations API. Something like this is quite nice:
``` myRect.animate( { "y": [0, 20, 0] }, { duration: 1000, easing: "ease-in-out" } ); ```
There's a polyfill for the burgeoning Web Animations API. Once Edge and FF implement these CSS props for SVG, it should be possible to seriously suggest doing at least some of the more straightforward D3 demos in plain JS.
But your point is well taken and getting going with polyfills where possible is a great idea. Things will be much nicer when this all gets filled out and consistent across engines.
Here are two tiny examples I've made with D3. I'd probably choose raw svg and a vdom lib next time, since I'm currently working on a project that mixes Mithril.js and SVG very nicely (although D3 or equivalent would be useful for mapping domains and scaling).
https://htmlpreview.github.io/?https://github.com/dsego/dseg...
https://htmlpreview.github.io/?https://github.com/dsego/dseg...
Now, as a DOM manip lib, I certainly like it a lot better than something like jQuery, but some of the newer offerings like React are a little more intentionally designed and don't have as many anachronistic design choices. D3 is just too much its own thing. There aren't many other JS libraries that follow the same DSL pattern that they've developed for it, so it can take quite a lot of ramp up time to get people used to it.
For example, I built a in-door navigation app. It would receive its data via MQTT/WebSockets and update markers on custom maps.
D3 was perfect for this, it helped with synchronizing graphics with data, converting mapping cooridnates to DOM/SVG coordinates and easing all the updates so stuff looked smooth.
You just would pump the incoming data into the D3 pipeline and be done with it.
If you just need a few animations, there are a bunch of simple libraries for this. React has react-motion for example, it's not as flexible as D3 but you get most stuff you would expect out of it without much hassle.
If you just need a set of basic charts, use a charting library. Highcharts or Charts.js are both very capable.
Even when compared with using C++ I have found using JavaScript based libraries difficult D3 in particular I found to be horrendously complex.
In a work context, D3.js is overkill most of the time. People expect line charts, maps, bubble charts that are simple and interactive...doing that in d3.js is a lot of copy/paste from examples and spending days to weeks working on simple details that would come OOTB in something like Highcharts.
I would only recommend d3.js if you really want to dive deep into the constructs of the visualization. If you want a vanilla chart, there's plenty of high level open source libraries you can use.
Unfortunately this is widespread across the JS ecosystem. Hell I'm not saying anyone should do it like Linux (break userspace and get shot), but when one does backwards-incompatible changes, is it so hard to stub dead methods or unsupported calling ways with "console.error('Outdated, see https://github.com/x/y/issues/z');"?
The complexity of designing a plot comes with how you compose together the basic elements. The consumer of the plot will want insets or annotations, two plots overlaid or juxtaposed. Error bars. It's very easy to get overwhelmed if your starting point is trying to design beautiful visuals. D3 will let you build all the needed complexity, but if you start by copying a beautiful plot, you'll soon find yourself overwhelmed, unable to deal with beauty and complexity at the same time, struggling to reconcile new functionality with the initial design.
My advice is to make the simplest possible plot of your data (one of the basic types, monochrome if you can get away with it), and figure out whether you need to add anything once you and your customer have tried to work with it.
It's true learning curve is different that productivity. Once you scale the mountain, if it's enough better, it's possible to be more productive in the long run. But this comes with a huge caveat - that there's not another way to design a system to get the same benefits in a more approachable way with a faster ramp up. Do a lot of people really believe that?
Even if there could be a better design it doesn't mean people shouldn't use D3. People use the best solution available however imperfect.
It's a bit ironic that the word complex is used below in describing how it's essentially not as complex as people think.
"The misconception...is that d3 and data visualization are the same thing. That to do data visualization one must learn and use all of d3...I like to emphasize that d3 is a toolkit for communicating complex data-driven concepts..."
A piece of cake! Here is a quantum game in D3.js: http://quantumgame.io/ :)
(full disclaimer: mine, and I already do too much of its shameless self-promotion)
Adding interactivity is then a matter of adding event handlers to those, but certain common use cases, say, moving the viewport, moving nodes, and drawing edges, can be tricky to get right for every edge (pun not intended) case and it can be easier to just use something ready-made.
It's source (Vue-based, potentially for other showcases/searches): https://github.com/wbkd/d3-discovery (I just asked if it can be open source, and to my awe its author added MIT license :)).
The real problem is this: data visualization is hard. UX is hard. I feel like a lot of people are looking for some magic-bullet framework that will let them achieve complex, highly-polished results without having to learn something.
Guess what -- there's no shortcut. You have to make an effort, learn, and practice. Most people will produce mediocre things at best, and most will give up.
Yes, viz can't become basket weaving. But there is always a spectrum on which things can be made more or less difficult.
Didn't some of your professors make subjects seem beautiful, elegant, easy to grok, while some of them seemed to make subjects no more inherently difficult inscrutable?
I stumbled on C3.js which is an abstraction over D3 for re-usable charts built atop D3. Never looked back. If anybody is in similar situation, please take a look.
Conflating these things is the biggest theme I see in the comments. I risk ranting here, because it seems like this conflation is a recurring misconception in our community. Maybe because most of us don't get to spend equal time thinking about design and usability in our specialties?
For the benefit of all the projects we both design and use, we must consider these as two important, yet separate concepts:
1) Any non-trivial concept, tool, or process requires serious investment to be productive with. We should expect it, not complain about it, and just roll up our sleeves if it's really the right tool for us and for the job.
2) We should never let any tooling or library off the usability or productivity hook because it's non-trivial. All designs/implementations do not have to be equal in this regard and are usually not.
Yes, we should be willing to invest our time and build the proper foundational knowledge. We should also always challenge how things can be made more productive, more approachable, and as simple as possible but no simpler. The guy to first think of that phrase worked with some pretty damn non-trivial concepts.
(similar to HighCharts, but without the commercial license mumbo-jumbo)
Yeah, I see D3 itself as more of a tool tailored towards DataVis geeks developing new visualisation methods.
Edit: The title of the stock photo is "Timmermansgereedschap" (NL) = Zimmermannsgerätschaften (DE) = carpenter's tools (EN).
SecurityError: Permission denied to access property "localizedStrings" on cross-origin objectI'm on FF 61 on Windows and it seems to work fine, same with FF 59 on Ubuntu.
This is my key takeaway that fully resonates with my way of learning anything related to JS be it D3, Angular, VueJS, Node, Android and more recently React. Take a problem, solve it in the tech of your choice.
First, I think if you want to use d3, you probably just want a charting library. Convince yourself that HighCharts is not flexible enough and then try d3. So the trouble with d3 is that most people don’t need to use it and are in over their heads.
Second, d3 is full of leaky abstractions. Imagine if cars didn’t have acceleration pedals, and you had to manually open the throttle and engage the fuel pump yourself. You’d need an engineering degree to drive your car! Same thing with d3; it’s ridiculous that you need to know the ins and outs of some of the internet’s most complicated and frustrating technology just to show a couple of data points on a graph. It’d be great if you could work with d3 and pretend like SVG and the DOM didn’t exist.
When I look at the examples at their page I see stuff that I could make in 2 ways:
- Just write a small program from scratch that visualizes one in a week or 2. It will be small, I will know the ins and outs, so easy to change and will be a ton of fun to write.
- Use D3.js (I dont know this library, but i've used complex charting libraries in the past). Likely it will take more time, not because I am thinking of the interesting parts, I will spend most of my day cursing on StackOverFlow to figure out how to skin that one tricky part of my app.
So in the end I dont really see a good reason to use d3.js unless my full time job is to develop data visualizations.
I still wouldn't say I've "learned D3", but I didn't have to. I got my goal accomplished in a couple of days, along with a lot of nice bits I didn't intend along the way as I found good things that D3 could do for me.
For me it was definitely a "design to tools" approach; I looked at the gallery, found something mostly like what I wanted, and then worked to that. That was fine by me; I wasn't looking for ultimate visual control, but just trying to turn a bunch of data into something mortals could comprehend.