An issue of visualization on the web I don’t know how to solve is how to load large data.
At Splitgraph [1] we're building a "data delivery network" (DDN, like a CDN but for databases) that looks like a big Postgres database. It works really well with the Observable Postgres client in private notebooks -- you configure it like you would for any Postgres connection. For public notebooks, you can use our HTTP API for sending SQL queries directly to the DDN. Here's an example [2] using that SQL-over-HTTP API to plot some Covid data.
[0] https://observablehq.com/@observablehq/connecting-to-databas...
[1] https://www.splitgraph.com
[2] https://observablehq.com/@mildbyte/splitgraph-ddn-oxcovid19-...
The idea is not surprising: using database indexes and a backend to send the frontend necessary data to render on demand. DB and the backend are containerized so the installation, data loading and authoring are all one command.
For creators, we offer D3 and JSON-based declarative primitives that enable creation of pan/zoom like visualizations very easily (e.g. 10s of lines of JSON for a 15-zoom-level vis). You can check out an interactive demo here, which visualizes 1.88 million wildfires: http://wildfire.kyrixdemo.live/
Since big data vis is very use case dependent, I'd like to reference two other tools that might be useful:
Falcon: https://github.com/uwdata/falcon NanoCubes: https://nanocubes.net/
Disclaimer: I'm still in academia so I'm apparently biased towards academic works. But fortunately they are all open source.
I checked out the Falcon documentation on Github and currently don't have a great understanding of (a) what it would be like to "write in Falcon" and (b) what it's intended use case is and how it differs from existing libraries.
Do you mind clarifying?
3 questions:
1. Why isn't there a demo I can just click and visit?
2. Why not TypeScript? (I think this would be a worthy short term and long term investment —for this project and beyond— to make the upgrade)
3. Why Docker? (This one I'm probably wrong about and am just a curmudgeonly old gray beard). Every time I see Docker I say "nope". I get it's use for massive clusters, but for running on one machine I dislike it (at that point I'd prefer just to run an image on a cloud VM).
1. In the original comment there is a link to a demo created by Kyrix: http://wildfire.kyrixdemo.live/ Are you looking for a different type of demo?
2. Honest answer: we are very understaffed academics who also need to write papers and theses. We want to convert to TS, that is just one item on the wish list.
3. We want everyone to be able to spin up a Kyrix app on their laptop using 3 CLI commands (you can try, instructions are in the README). With a backend and a database comes the cost of complex installation. Docker helps make everyone's dev environment consistent so it's easier to troubleshoot.
2. I get it. I never had the "pleasure" of being an academic, but got the chance to be a software engineer working alongside grad students for a couple of years and it felt like I imagine what taking a tour of a sweatshop factory floor would be like. (a huge exaggeration, but I saw a lot of low pay and lots of time on pdfs and not enough on code)
3. Yeah, I get it, but maybe allowing everyone to run it on their own VM would be better (and providing a one click "click here to get your own droplet on digital ocean running kyrix" sort of thing). I've found that Docker suffers from the XKCD problem (10 different environments to support—what if there was just 1....11 different environments to support!). I've almost never had a pleasant experience using Docker (and anything that requires downloading a 1GB+ image I don't consider pleasant). But again, I'm probably just being a grump and the other grads I worked with all seems to like it. Another idea is could use do this with SQLite? Have a more slimmed down version that didn't require Docker + those dependencies
Thanks for the answers! Very cool stuff and it's a very interesting problem.
An interesting thing to me here is the layering of DSLs. E.g., SQL enables user-defined functions like filters, that push down to multi-GPU columnar analytics with the rest of the pipeline... and GPU arrow dataframes for zero copy / streaming to combine it all together. People are posting in this thread about 1M rows, but this stuff is built for 1B+. The DSLs means both analysts and devs work at high levels, and underneath, supercomputing.
Fun historical note wrt JS vs Python for GPU: both have different strengths/weaknesses.. but are basically fine-enough long-term, with tweaking. We started w/ proving out JS GPU dataframes on OpenCL to be more open + viz friendly, and after Nvidia liked using our platform, their RAPIDS.ai team spun up to bring the idea in a more corporate controlled & funded way to Python. That's where the community resources are, so we jumped on board, and every month is now quite a trip. GPU SQL, GPU streaming, etc :) JS does inlining and async better than Python, while Python has the data ecosystem, so I've been eagerly anticipating JS folks stepping up where we had to leave off.
It's exciting to see it all come together -- imo, still early days for what's possible!
Btw, if you referred to the 1M-row example I posted - we actually can do much larger than that. A recording of a visualization with 1B reddit comments is here: https://youtu.be/ccES97ni_vI Behind the scenes we do indexing with Citus, which is a distributed version of PostgreSQL. Our cloud budget can only afford hosting a small demo 24/7 so that's why. Also, because OP talked about loading data into the web vis tools - 1M can already break many web tools out there.
You can tell our target use case is different than yours, and than OmniSci's. It's great to see solutions being developed in a one-size-doesn't-fit-all world.
- omnisci SQL. in contrast, rapids.ai opens layers below (cudf arrow, dask, ...) that enable cooperating solns on top (blazingsql, cugraph, custreams, prefect, ...) that are faster + easier for their domains, w fallback to general dataframes/sql.
-- omnisci is governance by a VC co, while rapids.ai is by nvidia (who wants to sell hw, not sw) and more OSS partners
Omnisci did good engineering, so it does have strengths. ex: its geospatial visual analytics means it's a good esri alternative consideration, as it is more polished than manually stitching together cuspatial + blazingsql + leaflet etc. Likewise, commercially polished for hostile enterprise environments (procurement, ...).
re:scale, see rapids tpcx-bb numbers ('big data'), I think on 10TB datasets. it shows scale + cost effectiveness wins vs others. less obvious, out-of-core so can do TBs even on one GPU, and full tpcx-bb needed the above versatility where sql is a kludge.
re:graph vs table, if you do just points and no edges, the node table is just a regular table you can do regular tabular data analysis + viz in. ex: load in samples scored by some ml model (x/y plot w lots of data columns for each point), then connect nearest neighbors to make it into an interactive graph. we are doing more and more here in practice, it's fun :)