Show HN: Plotting 3 years of hourly data in 150ms
leeoniya.github.io
leeoniya.github.io
A small part of the problem is drawing the trend charts. So I decided to make uPlot [1] to see what was really possible.
a lot of time is spent on just DOM layout and JITing the JS, which get amortized with huge datasets. the 166k bench doesnt take any longer.
The source code, which can be used as a Dash + Dask boilerplate, is here: https://github.com/plotly/dash-world-cell-towers
Feel free to ask questions at community.plot.ly
Credibility?
>The range field tells you the approximate area within which the cell could be, in metres. This is effectively an estimate of how accurate the location is.
There's one marked in the middle of a corn field near me with a range of 48,000m. ?
Station: https://carlisleweather.com
Software: https://github.com/chrissnell/gopherwx
I have done some visualization with WebGL because I couldn't get it fast enough with just drawing lines. That was a while ago so I'm not sure about the details, but even a simple prototype just drawing a few tens of thousands to a hundred thousand lines using canvas was slower than this for me (subjectively).
I'll have to look at the code later, but I'm curious about where this library is getting the performance from. Before I saw this I would have said you can't do this at this speed without using previously downsampled version of the data points, I'm not entirely sure now.
I'm still looking at the performance tab in the browser dev tools and how impressively empty the main Javascript usage plot is.
The selection highlight is unusual, I admit. That's something I'd just skip if I were going for high performance.
no, the top graph is pretty much what the title says: 3 * 365 * 24.
but 194k is also no problem :)
you're right, it isn't. and yet many js charting libs struggle even with this.
on my i5 thinkpad with integrated gpu uPlot can render 600 series x 8,000 datapoints in ~2000ms [1]. and it finishes this job in ~7s on my 2015 sony z5 compact phone. so there's that :)
but really, pumping 4.8M datapoints into a browser is probably not a great idea. you're gonna want to aggregate on the server at some point. just sayin'.
[1] https://leeoniya.github.io/uPlot/bench/uPlot-600-series.html
Just want to ask, because I love talking about render performance: have you tried doing this using offfscreen canvas? That should allow you to move a lot of things to a worker, so you avoid blocking the rendering with js? It probably won’t speed up the total time to finished render, but I assume it will lock the page for a shorter period of time?
- the data is flat arrays of numbers
- there is no per-datapoint memory allocation beyond whatever is necessary for the browser to construct the canvas Path2D object. this keeps the memory pressure low and the GC nearly silent.
- the amount of draw commands is reduced by accumulating the min/max data values per pixel
- uPlot does not generate axis ticks by walking the data. it only uses min/max of the x and y ranges and finds the divisions from that.
- there is no mass-creation of Date objects, data is kept in timestamp format except when hovered.
- the date/time formatting is done by pre-compiling templates and not re-parsing them all the time.
- the cursor interaction uses a binary search over the x array
- drawing at exact aligned pixel boundaries to avoid or minimize antialiasing
this makes uPlot charts a bit rougher looking, but the perf impact is quite large.
I prefer the look of things snapped to pixels; I didn't realize it also speed things up!
it's highly dependent on what's actually drawn. i regressed it by accident and didnt notice a huge difference until i randomly opened the stress test (which is also densely packed so probably a worst case for AA)[1]. it went up by a factor of 2-4. cant remember exactly.
[1] https://github.com/leeoniya/uPlot/blob/master/bench/uPlot-60...
When we talk (or rant) about the performance of JS-based apps - running on the web or in Electron or otherwise - it's really important that the conversation focus on the factors that actually matter.
The fastest code is no code. No code is fast even with pathologically slow languages/runtimes.
But that wasn't my main point: my main point is that people love to rant about "JavaScript apps" while maintaining (willful?) ignorance about what the actual factors are that manifest as the slowness they experience. Sometimes it's poor usage of the DOM. Usually it's ads. It's almost never the unavoidable overhead of JS cycles.
Are you sure you got this right?
Just because JS is fast[0] doesn't mean we cannot complain about Javascript apps that should have been plain web pages?
It is clearly possible to create advanced Javascript apps that are enjoyable even to people like me.
For most pages I use as a consumer I don't see much value add because of frontend code: I see the value of autocomplete, drag-and-drop etc but I would much prefer if my CPU stopped chewing after web pages were loaded and rendered.
[0]: yes, I agree - in most cases the Javascript language or engine is not the cause of the performance problems on the web
this happens when it's programmed with performance as an afterthought, which is sadly how things work when the prerogative is to just ship more features and continually ingest dependencies with the most features, with little vetting.
Ad bloat is a valid discussion
Lazy development practices in modern sites/apps are a valid discussion
But in my experience, these as well as other less-legitimate issues all tend to get lumped under the banner of "JavaScript stuff == slow", without any nuance.
I guess Javascript just happens to be the common thing between a number of them, and people would be just as annoyed if the multi-megabyte, cpu-hogging, data-stealing monstrosities where hand-crafted in assembly ;-)
And: If someone complained about Javascript being slow for scientific calculations I'd probably call them out on it ;-)
Question: in the demo you load papaparse but the data is delivered as JSON?
Would you mind sharing what your backend looks like for the project where you're using this, and how do you extract all of this data at once from storage?
good catch, leftovers from original PoC i made locally. the exported data was still in CSV back then. i should clean that up!
> Would you mind sharing what your backend looks like for the project where you're using this, and how do you extract all of this data at once from storage?
i just did a manual export from Analytics. how you get the data out is up to you, but a time-series or columnar database would probably be a good start :)
The only thing I wish µPlot had is support for tooltips rather than numbers in the legend, but that's a small price to pay for this level of performance.
[1] https://leeoniya.github.io/uPlot/demos/candlestick-ohlc.html
lol :(
336524=26280. Emitting 26k points is like child's play for any program, are we really so far down in bloat that this is considered impressive? Because this is not impressive. A 16MHz AVR could execute 91 instructions per sample in those 150ms, more than enough for plotting that data.
A 3GHz machine could execute 17123 instructions per sample, and it would probably need less instructions than the AVR would.
try to get this perf on the web, and then you can re-asses your statement. if it was easy, then every other js charting lib would not be struggling to do it, right?
for native code (or webgl), obviously this is child's play. but webgl has significant trade-offs. e.g: https://bugs.chromium.org/p/chromium/issues/detail?id=771792
finally, you cannot extrapolate from the 26k/150ms number. uPlot can draw 4.8M points in ~2000ms on an i5 with integrated gpu (after bootup amortization).
I was slightly disappointed when I zoomed in as I expected 3 years of data in 150ms increments (600 million measurements) but that's probably not possible
Could you compare your performance against this one if possible?
Great job btw, really instant.
Either way, just a bit of feedback on a beautiful page
I work with analytic tools to watch production systems and they boil my data down to a pathetic number of datapoints prior to rendering, when you know computers — even in a browser — are vastly more capable.
That said, things like SignalFx or Datadog are certainly slow at the presentation layer, too ;)
If there was a very pronounced weekday effect, a quick hack is to use a 364-day instead of full-year offset for comparison.
Here's a sample chart with 1,000,000 points: https://jsfiddle.net/5bvLgs5w/1/
https://github.com/leeoniya/uPlot/blob/master/bench/Highchar...
additionally, it's hard to overstate how impressive modern javascript JITs are, as well as GPUs.
every time i visit a web page that downloads 2MB of js and spins up my CPU fan, i feel ashamed that this is the industry (webdev) where i make my living.
at least with uPlot i show that all this waste is not necessary.
You could probably sample all the 1D textures in one pass, and draw all the lines and fills there. One additional nice side effect of this is that you can easily have sequences at different resolutions.