Comment by kuatroka | Hacker News Reader

Comment by kuatroka | Hacker News Reader

A couple of questions:

1. Let's say I got a Sqlite/Duckdb database file on my server. It's got multiple tables and some of them 100M to 150M records. I want to create a plot/table that would have a slider/filter to only bring and show a slice of data at a time. Since it's statically generated data, how is this interactivity achieved? All the possible facets of data filtered by which ever way will be generated? Won't it be huge and how long will it take to generates this static data or is there an actual call back to the server to the duckdb file (I assume it works with .duckdb file too?)

2. If Observable Framework provides the front-end, does it mean I can use any auth library if I want to create a web site with a log in and subscription options?

3. If it's a static web page, does it mean that at any time a user views a chart, they will also be able to go to the Dev Tools and download the file with data that's behind the viz?

4. When (if you can share of course) is the planned release of Plot's interactions: zoom, pan, interactive legend, brush?

5. Deployment - with big parquet, sqlite, csv files, it's impossible to do CI/CO through github or vercel and such. Will your hosting services offer an option to host those files and runtimes to generate them?

Thanks

Good questions.

1. It’s just JavaScript so you can fetch stuff dynamically too (see https://observablehq.com/framework/lib/duckdb). But yeah, only client-side. (Though see https://github.com/observablehq/framework/issues/234.)

2. Sure, it’s all open source, I bet you could make that work. Or `yarn deploy` to Observable and configure sharing there (though it wouldn’t let you charge others).

3. Yup. Which is part of the appeal of model of running data loaders at build time: you can query some private data and viewers would only be able to see the final result set. (The lack of something like this has always been a huge problem for Observable notebooks. You’d make some great query-driven charts and then couldn’t make it public without some awkward manual dance of downloading and re-uploading a file to a fork of the notebook.)

4. I wish I knew! It’s being tracked here https://github.com/observablehq/plot/issues/1711. Lately there’s been a lot more work on Framework naturally but now that that’s out…

5. Another good question. We’re definitely interested in tailoring it more to this sort of use case but lots is TBD!

Thank you I wonder if for #3 there is a way of somehow to keep the data hidden and only let people see the chart without hacking their way to see the underlying data

During our early exploration, someone made a data loader that returned an entire svg of a chart, instead of the data for a chart. I think it was a headless browser running Observable Plot, but I imagine there are lots of ways to generate charts in a data loader.

Came here with similar questions and Cmd-F "DuckDB". See the comment about "data loaders". Seems like a "data loader" would provide most of what you're asking about.

I'm also thinking that a "data loader" combined with duckdb-wasm and arrow would be a pretty nice combination. I imagine that it might not be too difficult to switch two between two implementations of the "data loader" as needed. Switch between reading from a remote system (in your case DuckDB on a server) and DuckDB running locally in the browser (that can interact with its own remote or local data sources).

edit: welp https://observablehq.com/framework/lib/duckdb

See the example at https://huggingface.co/spaces/observablehq/fpdn where DuckDB is used both as a data loader (to download and digest 200GB worth of source data into a small 8MB parquet file) and on the client-side to allow the user to do live search queries on the minimized data. Server-side, we're using duckdb-the-binary, and client-side we're using duckdb-wasm.

So the 200Gb loading and digesting part is totally separate from the Observable Framework, right? You just do it with a standard ( non wasm duckdb as part of ETL) and later you just direct Observable Framework to read and plot the 8Gb file? Thanks

nope, Observable Framework data loader accesses the 200GB dataset. The code is here: https://huggingface.co/spaces/observablehq/fpdn/blob/main/do...

Reply on news.ycombinator.com