Hacker News Reader

Comment by tracyhenry | Hacker News Reader

We are building a data vis tool (https://github.com/tracyhenry/kyrix) to solve this exact problem many web vis tools have.

The idea is not surprising: using database indexes and a backend to send the frontend necessary data to render on demand. DB and the backend are containerized so the installation, data loading and authoring are all one command.

For creators, we offer D3 and JSON-based declarative primitives that enable creation of pan/zoom like visualizations very easily (e.g. 10s of lines of JSON for a 15-zoom-level vis). You can check out an interactive demo here, which visualizes 1.88 million wildfires: http://wildfire.kyrixdemo.live/

Since big data vis is very use case dependent, I'd like to reference two other tools that might be useful:

Falcon: https://github.com/uwdata/falcon NanoCubes: https://nanocubes.net/

Disclaimer: I'm still in academia so I'm apparently biased towards academic works. But fortunately they are all open source.

View on HN

Vega-Lite and Falcon author here. Many people ask for scalable visualizations and I think declarative visualization approaches are really promising since systems can transparently optimize data movement and rendering. I have done some more experiments to scale Vega such as https://github.com/vega/scalable-vega but I think what we need is a system that automatically offloads heavy computations to a backend such as https://github.com/omnisci/jupyterlab-omnisci.

I've used Vega-Lite and d3 before and have appreciated both. Vega-Lite seems to be great for rapid prototyping and d3 for really refined, intricate and more complex plots.

I checked out the Falcon documentation on Github and currently don't have a great understanding of (a) what it would be like to "write in Falcon" and (b) what it's intended use case is and how it differs from existing libraries.

Do you mind clarifying?

This feels close to a lot of our thinking and why some of those pieces were originally written :) We (the Graphistry team) wrote the original Arrow JS implementation to help us bridge GPU visual analytics components in the browser with real-time GPU clusters in the data center, and our backend is DSLs like dataframes to make it easier to do that.

An interesting thing to me here is the layering of DSLs. E.g., SQL enables user-defined functions like filters, that push down to multi-GPU columnar analytics with the rest of the pipeline... and GPU arrow dataframes for zero copy / streaming to combine it all together. People are posting in this thread about 1M rows, but this stuff is built for 1B+. The DSLs means both analysts and devs work at high levels, and underneath, supercomputing.

Fun historical note wrt JS vs Python for GPU: both have different strengths/weaknesses.. but are basically fine-enough long-term, with tweaking. We started w/ proving out JS GPU dataframes on OpenCL to be more open + viz friendly, and after Nvidia liked using our platform, their RAPIDS.ai team spun up to bring the idea in a more corporate controlled & funded way to Python. That's where the community resources are, so we jumped on board, and every month is now quite a trip. GPU SQL, GPU streaming, etc :) JS does inlining and async better than Python, while Python has the data ecosystem, so I've been eagerly anticipating JS folks stepping up where we had to leave off.

It's exciting to see it all come together -- imo, still early days for what's possible!

Thanks for the thoughts. Graphistry looks really slick and interesting. On thing I'm curious about: how do you compare Graphistry with OmniSci and similar GPU-based solutions? You seem to focus on a niche - graph visualizations. That's one differentiator I can tell. Do other solutions support graphs too? If so, how do you compare Graphistry with them?

Btw, if you referred to the 1M-row example I posted - we actually can do much larger than that. A recording of a visualization with 1B reddit comments is here: https://youtu.be/ccES97ni_vI Behind the scenes we do indexing with Citus, which is a distributed version of PostgreSQL. Our cloud budget can only afford hosting a small demo 24/7 so that's why. Also, because OP talked about loading data into the web vis tools - 1M can already break many web tools out there.

You can tell our target use case is different than yours, and than OmniSci's. It's great to see solutions being developed in a one-size-doesn't-fit-all world.

Fundamentally, omnisci locks you into:

- omnisci SQL. in contrast, rapids.ai opens layers below (cudf arrow, dask, ...) that enable cooperating solns on top (blazingsql, cugraph, custreams, prefect, ...) that are faster + easier for their domains, w fallback to general dataframes/sql.

-- omnisci is governance by a VC co, while rapids.ai is by nvidia (who wants to sell hw, not sw) and more OSS partners

Omnisci did good engineering, so it does have strengths. ex: its geospatial visual analytics means it's a good esri alternative consideration, as it is more polished than manually stitching together cuspatial + blazingsql + leaflet etc. Likewise, commercially polished for hostile enterprise environments (procurement, ...).

re:scale, see rapids tpcx-bb numbers ('big data'), I think on 10TB datasets. it shows scale + cost effectiveness wins vs others. less obvious, out-of-core so can do TBs even on one GPU, and full tpcx-bb needed the above versatility where sql is a kludge.

re:graph vs table, if you do just points and no edges, the node table is just a regular table you can do regular tabular data analysis + viz in. ex: load in samples scored by some ml model (x/y plot w lots of data columns for each point), then connect nearest neighbors to make it into an interactive graph. we are doing more and more here in practice, it's fun :)

I saw you mention Falcon and NanoCubes so had to check our kyrix. I think this is a really important space. I used to assume Palantir had solved this but now that they're public it seems to be more of a smoke and mirrors startup.

3 questions:

1. Why isn't there a demo I can just click and visit?

2. Why not TypeScript? (I think this would be a worthy short term and long term investment —for this project and beyond— to make the upgrade)

3. Why Docker? (This one I'm probably wrong about and am just a curmudgeonly old gray beard). Every time I see Docker I say "nope". I get it's use for massive clusters, but for running on one machine I dislike it (at that point I'd prefer just to run an image on a cloud VM).

Thanks for the feedback!

1. In the original comment there is a link to a demo created by Kyrix: http://wildfire.kyrixdemo.live/ Are you looking for a different type of demo?

2. Honest answer: we are very understaffed academics who also need to write papers and theses. We want to convert to TS, that is just one item on the wish list.

3. We want everyone to be able to spin up a Kyrix app on their laptop using 3 CLI commands (you can try, instructions are in the README). With a backend and a database comes the cost of complex installation. Docker helps make everyone's dev environment consistent so it's easier to troubleshoot.

1. Ah! Sorry I missed that and went to the GitHub. Might be helpful to also add that link to the "website" field of the GitHub?

2. I get it. I never had the "pleasure" of being an academic, but got the chance to be a software engineer working alongside grad students for a couple of years and it felt like I imagine what taking a tour of a sweatshop factory floor would be like. (a huge exaggeration, but I saw a lot of low pay and lots of time on pdfs and not enough on code)

3. Yeah, I get it, but maybe allowing everyone to run it on their own VM would be better (and providing a one click "click here to get your own droplet on digital ocean running kyrix" sort of thing). I've found that Docker suffers from the XKCD problem (10 different environments to support—what if there was just 1....11 different environments to support!). I've almost never had a pleasant experience using Docker (and anything that requires downloading a 1GB+ image I don't consider pleasant). But again, I'm probably just being a grump and the other grads I worked with all seems to like it. Another idea is could use do this with SQLite? Have a more slimmed down version that didn't require Docker + those dependencies

Thanks for the answers! Very cool stuff and it's a very interesting problem.

Again, thanks for the feedback. Your comments are great food for thought!

Generally Docker lets you say 'here, this works', rather than 'follow these 800 steps to get your environment in the right shape'.

And now your env is in this weird state that doesn't work for the previous or next tool you try.

Reply on news.ycombinator.com