Time Series Density Plot | Hacker News Reader

Time Series Density Plot

observablehq.com

40 pointsluu16 comments

so... a finely-binned colormap?

I'm not sure it's not quite right either, because the weight of each series in a bin should depend on the derivative but it doesn't look like it does.

This sort of plot always has a conflict between showing the distribution of each time domain sample and showing the time behavior of each curve. For the former, showing quantile contours may be clearer (though not as pretty!). For the latter different colors with transparency might be better, though it becomes unintelligible at some point.

The weight does depend on the derivative: https://arxiv.org/abs/1808.06019

(This is mentioned in the intro post for this library: https://observablehq.com/@twitter/density-plot-introduction?...)

hmm, it's not obvious (to me!) in the single-line case, but that might be because of the high-frequency noise that makes the derivative sort of random.

Here's a quick fork with one line, no noise, and an increasing-frequency sine wave: https://observablehq.com/d/ed17fc04bb7eb887

Ah cool!

I suppose one can incorporate uncertainty on f(t) by having g(y,t) = gausy(0,\sigma(t)) * \delta(y-f(t)) (which would also effectively antialias in the few-lines case).

Also I guess you have to be careful not to plot a Weierstrass Function!

Wasn't immediately clear to me why the derivative comes into play, but let me see if I understand the gist of it. I think this is something interesting I haven't thought about in dataviz before...

So in a traditional 2D heatmap, we discretize the space into squares or bins, then color the bin "proportional" to the number of points that fall inside the bin.

If we're making a 2D heatmap of a timeseries, though, we don't really want to count up the discrete points that make up the timeseries because our binning may not line up with our timeseries sampling frequency. Especially not if we want a very high-resolution heatmap, where each bin is sized to be 1 pixel.

So the solution isn't to count the number of points (from the timeseries data series) in each bin, but rather to count the number of lines that get projected/drawn into each bin (pixel).

But we need to go a step further... if we think of each bin as a hypothetical square, the shortest line through a square is a vertical or horizontal line. And the longest line through a square is a diagonal line. So if we want to represent "how much" of the line goes through a square, we need to measure its slope. Hence, derivative.

So by 2D heatmapping lines instead of points, we're not just counting how many lines fall into each bin, but we also need to weigh each bin-line occurrence by it's arc length, which we use the derivative as a proxy for.

Neat! (I think...)

I like to think about it from the perspective of continuous time, which is approximated by having #bins = #pixels. The approach relies on having fine-grained bins, since they make error arising from the relationship between binning and the sampling frequency much less significant.

Imagine a point moving along the curve, depositing a constant amount of density/ink onto the canvas per unit of time. When the point is moving quickly it deposits less density, and when the point is moving slowly then it deposits more, since the density deposited per time unit is constant and the point traverses less space when it moves slowly.

You can think of each vertical strip of bins as representing a unit of time. The discrete approximation to arc-length normalization means, for each time series, making sure that it contributes a single unit of density per vertical strip: if a curve goes through one bin in that strip, then that bin has its density increased by 1. If it goes through 3 bins, then the density in each is increased by 1/3.

Ah, so the quick explanation is how many bins per unit time does the line hit (how many pixels in a vertical stack). If the line is horizontal and hits one bin, then it should have a weight of 1. If it's steep and hits 3 or 5 bins, then the weight should be 1/3 or 1/5 for each bin-pixel.

That makes sense. Thanks!

"Here's how to..." call a function in a library that does all the work for you.

Expected a bit more.

This page was specifically created as a library example. Here's the full collection: https://observablehq.com/collection/@twitter/density

If you're interested in the technique behind the plot you can read more about how that works in this paper: https://arxiv.org/abs/1808.06019

We use a homegrown gnuplot version of this at $work, and I can speak from experience that you can immediately spot problems (timeouts, as a set of ghostly horizontal lines, or outlier requests) that would be absolutely hidden in a traditional p50/p95/p99 chart.

I wish more (hosted) monitoring systems would shift to this type of visualization, since computers are fast now (I'm told!)

Aren’t these spaghetti just plots with transparency. Or am I missing something? Another nice way to illustrate density are fan charts. While spaghetti plots show individual time series, fan charts just show quantiles. Spaghetti plots therefore also reveal information of the conditional distribution of a time series which fan charts do not. The latter are just easier to read.

Hi, author here. You can think of density plots as a way to visualize the data traditionally visualized with line charts or spaghetti plots, but in a way that scales more effectively to large datasets.

Traditional graphics pipelines aren't built to handle more than 256 incremental levels of opacity since it's an 8-bit channel, and when you have a thousand time series you can no longer distinguish between levels of density. Conditional information (following a single line) becomes difficult as well since there are so many lines. Density plots solve the overdraw problem by accumulating density into an offscreen buffer before rendering, which allows the color scale to be tuned to the amount of overlap in the specific dataset. Interactive selection/hover techniques can be used to recover conditional information.

This example is part of a density plot library published earlier this week (https://observablehq.com/@twitter/density-plot-introduction?...) and one of the references there is to an opacity-based approach I've seen used for 2D point clouds, though I haven't seen an analogous demo for curves: https://observablehq.com/@rreusser/selecting-the-right-opaci...

Nice work! Density plots aren't just pretty. I think they show a more accurate story for any time series plot with more than a few thousand points.

The reduceOp caught my eye in your API. What uses have you imagined/tried for changing the density accumulation function from x + y?

Thanks! I've used it when the data is itself binned frequency data, which is one way to reduce data volume (e.g. binning 2 weeks of minutely time series data into (hour, percentile) buckets and plotting them as weighted points).

When the number of the bins in the data is not an exact integer multiple of the number of bins in the histogram, adjacent data bins can get mapped to the same histogram bin, resulting in e.g. 2x the data volume in some rows/columns of the histogram.

When a bit of loss of fidelity is acceptable the two solutions I've used are to render the histogram at an exact factor of the number of data bins then set the canvas dimensions to the desired size (relying on the browser to downsample the resulting image), or to use `max` as the reduceOp and render the histogram directly at the intended size.

One of my favorite plot types. Very simple and immediately conveying the necessary information. My colleagues in the lab dislike it, though, they prefer something more traditional with error bars, as they are more quantitatively inclined.

Reply on news.ycombinator.com