D3 in Depth | Hacker News Reader

D3 in Depth

d3indepth.com

493 points_mhr_52 comments

I think this hasn't been update since 2017, and most of it is from 2016.

I pretty much only use javascript for d3, and I put it down for a while. Recently I decided to do something with d3 v5.0[1], and got confused about the state of ES6 modules. Has anyone gotten rollup to work with d3? Is it possible to just use ES6 modules natively in the browser if you don't care about older browsers?

My other issue was with async/await. It seems like when I used async await on a locally hosted csv file, it could take between 1 and 2 seconds to load. For my colleague sometimes minutes. When I switched to d3 v4.0 and the old callback approach, it loaded instantly, even for my colleague. What gives?

[1] https://github.com/d3/d3/blob/master/CHANGES.md

Most ES6 features are available in Chrome and Firefox. Not sure what the issue is with async and await, all it is, is syntactic sugar really. It shouldn't have a lag time any differently.

Babel (transpiler) is super easy to setup if you have any problems with compatibility.

It's not all sugar, there are some actual differences; and especially around async/await there are optimizations that can only be made in the non-transpiled code.

He said async/await is syntactic sugar over promises which it is. And that babel is easy to setup, which it is. Those statements are independently true. No one said the state machine that babel transpiles your async/awaits to is performant, he wasn't making that assertion.

The difference between 1s & 1m lag in D3 has got to be a D3 bug.

> Is it possible to just use ES6 modules natively in the browser if you don't care about older browsers?

Yes. Note that you need to use

    <script type=module>

Personally though, I am using Google Closure compiler to turn the code of all my ES6 modules into a single, packed file without modules. Additionally I have told Closure compiler to output a sourcemap so that I can debug my code still. There are some issues with this from time to time. Not sure about what the debugging story is like currently with plain ES6 modules in the browser.

Here is my Makefile in case you'd like to look at the arguments I pass to Closure compiler: https://github.com/ctsrc/Infinite-Tower-Run/blob/191a4035fd1...

For me using D3 is kinda like using regex. Whenever I need to use it... it always take a long refresher to get back up-to-speed. But, it's one of those amazing, invaluable tools that does some pretty complex stuff without sweat.

Really good comparison, feel the same way. I use a site to brush up on my regex now and again, but since I'm not using daily, its easy to forget it!

Which site are you using to refresh your regex?

I use this one https://regex101.com/

I really like http://regexstorm.net/ for .net based regex- the included regex tester is a very nice feature to debug a regex...

https://regexr.com saved my life

In addition to the online tools that other commenters have pointed out, I strongly recommend finding a regex "cheat sheet" (there are dozens online) and actually printing it out - I have one pinned to the corkboard next to my monitor and it is invaluable. It will save a lot of time from having to remember all the character classes, which characters do/don't need an escape \, and crazy lookaround syntax ("was it ?<! or ?<=")

I'll add Debuggex to the list that does a visual representation of a Regex and can be a god-send. https://www.debuggex.com

or ggplot

I love D3 but my biggest complaint is that many of the examples floating around appear to have been coded by people who don't care about reusability (and in some cases, ability for others to understand the code).

Nowadays if I need to do interactive visualisations I typically end up using Mobx + SVG.js. It works really well and I rarely get other developers asking me to explain how it works so that's a HUGE plus.

Maybe you'd like this library: https://github.com/curran/d3-component

It let's you modularize your d3 code into components. Very handy!

thanks! Bookmarked.

As a data person, I've always wanted to use D3 to build my visualizations but it's always been like using Assembly to build a program.

that's pretty much the only way to do it though.. if you go any higher-level then it's just pre-baked out of the box visualization. As low level as d3 seems to be, there is probably no more intuitive data visualization that is bound to web display technology.

This seems so defeatist. Is each graph really that much of a special snowflake that we can't have frameworks that cover the 80% use case but allow you to drop to a lower level when needed?

I am disinclined by temperament to use low level tools for tasks that really should have a sensible generic abstraction. I know some people's brains err in the other direction - each to their own.

(I'm pretty sure tools exist. I used https://bokeh.pydata.org a few years ago when I needed something along these lines. I looked at d3 and had a similar reaction to the grandfather comment)

There are higher level visualization tools built on D3.

I agree with he sentiment as an outsider, but let's build 5 bar charts and talk about this stuff again :-)

I guess I'm one of those people who thinks that d3's api is an amazing level of abstraction. It took a bit for it to click for me, but that was mostly because it was my first exposure to functional programming.

If I ever have to make a chart with another language, I cry because I can't use d3. I have reimplemented portions of d3's api in other environments like native Android before. Digging through the docs to find whatever configuration variable I'm supposed to set reminds me of the days of jquery plugins.

I will say though that it's been a side-project idea of mine for several years now to build a JSX-like templating language that would generate d3 code. Never got around to it.

Can you point me to some resources that can help me with D3?

I'd be happy to. Can I ask first what you've tried and what you had trouble with? I think a lot of why d3 is considered low-level is that there are lot of external things you do need to know. For example, for me, it wasn't d3 itself but functional programming that I didn't really understand. For those who don't do web development much, it might be DOM (and in particular, SVG) manipulation.

The d3 specific parts are the idea of mapping data to DOM elements (which is the part I think is beautiful), how it uses the data join to do so, and how you use the scale functions to map from data space to pixel space to set the element positions and other attributes.

Feel free to email me if you like -- it's on my profile.

If what you want is a tool that basically does Excel-style static charts for data, then d3 is absolutely the wrong tool. Where d3 shines is when you need to handle smooth, dynamic updates.

In practice, handling dynamic updates requires so many hooks into the data that you can't really define a sensible abstraction layer and have it work well for both simple cases and complex cases. D3 is the successor to Protovis, itself the successor to Flare, itself the successor to Prefuse. And having used all of them save Flare (since I've never had the ability to write Flash code nor the desire to do so), d3 is the only one that I've actually enjoyed using.

You can still use d3, but use a d3 charting library on top of it like nvd3 [1]. It gives you a prebuilt chart but you can modify it using d3 commands. It’s pretty useful and high level, but if you need the low level api it is there.

[1]: https://nvd3.org

As far as using other abstractions, you're at the mercy of the developer(s) who made the abstraction, and I've found having to do lots of extra work to figure out how to inject desired customization that makes it generally worth it to just use d3 in the first place

Yeah, it basically is.

I played with D3 every now and then for maybe the past 6 years to scratch some data visualization itch, but I never really understood it. I always wound up trying a code snippet from a random blog and got frustrated when I couldn't get it to bend to my needs.

It finally clicked for me a after I read the first few introductory tutorials on the D3 Github wiki. It made me realize that, while it was probably built for data visualizations, D3 at its core is a browser DOM manipulation library with a (data-driven) declarative API. It's a slightly higher level of abstraction compared to the DOM manipulation part of jQuery (though, many frontend developers these days consider jQuery to be a low-level library).

Of course, it also helped that I learned a lot over 6 years. I was a university student back then, and now I'm a professional software engineer who has spent the last year or so doing frontend development. D3 makes a lot of sense within the context of my current understanding of how browser DOM APIs work.

I was about to ask the question, does anyone use D3 directly?

I was under the impression that a lot of higher level libraries were built on it.

Yes, I used D3 directly everyday for four or five years as did the whole graphics team of the Financial Times where I worked. Most of the print edition charts were created via D3 at some point in the process too. If you're making datavis everyday there's not much to compare (ggplot is good but much of the time the visual quality of the output is not quite there). "Higher level" libraries i.e. those that impose a taxonomic view of visualisation tend lack the necessary flexibility and expressiveness.

Nice timing. I just found this website the other day and love it. I've just recently been getting into data visualization and have fallen in love with d3.js. Another good resources is http://alignedleft.com/work/d3-book/. Maybe a bit introductory for some of the smartypants on HN but I've been away from front end dev for a while and so find blasting through the chapters pretty helpful. Hilarious tone, also. Combined with d3indepth.com's examples and the API docs I've been able to get a good grasp of d3.js.

As someone who mostly sticks to python but is interested in data analysis and visualization, I still haven’t figured out what problem d3 solves, and which use cases it makes sense for?

What a strange comment. Have you tried to answer this for yourself? For example, looking at the homepage or any demos?

It offers a bunch of functions that help make svg manipulation digestible. Try manipulating an svg with vanilla javascript, it's a pain to do anything more than basic shape drawing.

data visualization that is interactive and works on a platform that is universal. python produces only static output right?

What? No. Check out Jupyter, for example. Of course that's a totally different platform, but I wouldn't call that "static".

but you need a special viewer to run the output.. huge friction. That's a none-starter for almost everyone.

everyone has a web browser...

...The Jupyter Notebook is an open-source web application that allows you to create and share documents that contain live code, equations, visualizations and narrative text.

http://jupyter.org/

Calling it "live" seems just slightly inaccurate to me. Every time code needs to run an http request needs to be made. (Unless there's some new tech here I'm unaware of)

This is a limitation that may make no difference in some cases, but in others it is important for code to execute in the client.

A good example of this is making a visualization that seamlessly responds to the user's mouse pointer.

But it is static. The visual output is usually in jpg form. That's pretty much the definition of static output.

The only output I can find in Jupyter that is not static is coming from Bokeh which takes Python and generates javascript code to run in the browser. Jupyter itself calls this "a style of D3".

Jupyter is just the editor and viewer. It can plug in a lot of stuff, including maps and videos, but those are prepared using other libraries.

https://bokeh.pydata.org offers a nice dynamic chart library from Python. It has a js helper library to handle to dynamism.

https://bokeh.pydata.org/en/latest/docs/gallery/hexbin.html

Write Python, make interactive web plots.

You can’t run Python (or R, or SAS or whatever) in a browser.

If you want to do dynamic visualisation with something like Python or R in a browser you need need a server, and that incurs significant latency.

D3 also allows much more fine grained control over visualisations than any visualisation library or package I’ve worked with in any language, because you have total control of every line, point, every fill.

No matter what crazy design the design team comes up with for your website, D3 will let you bind data to it and turn it into a workable chart.

It was written by an editor at the New York Times, and it is designed for producing bespoke visualisations on public facing showcase pages where appearances and speed have to be state of the art.

> If you want to do dynamic visualisation with something like Python or R in a browser you need need a server, and that incurs significant latency.

Not true. Bokeh (http://bokeh.pydata.org) lets you build interactive web graphics that run purely embedded in a static HTML file. No server needed.

(Of course, it also has a server-based mode that lets you build interactive analytical viz apps, with browser interactions automagically driving Python callbacks)

Try making this in Python:

https://archive.nytimes.com/www.nytimes.com/interactive/2012...

d3 is for interactive charts

You mean like https://demo.bokehplots.com/apps/gapminder ?

I hope this makes sense:

It's not so much for people wanting to make new 'instances' of a visualisation. It's more for people wanting to make new 'classes' of visualisations.

Or to try and explain that better... It's not a chart library, but a framework for wiring up dynamic data to collections of DOM nodes (either HTML or SVG) in any way you like. eg you can write specific chart libraries using it.

It is oriented around the lifecycles of instances of data - eg entry and exit hooks to run as data appears and disappears, and it has a lot of powerful lower level functionality.

This slide has a nice answer:

http://mbostock.github.io/d3/talk/20111116/#8

Reply on news.ycombinator.com