Hacker News Reader

Comment by 0cf8612b2e1e | Hacker News Reader

Sounds a bit like the “baked data” pattern. Which I think is a really good idea. I have long been toying with how to use Datasette to make a deliverable dashboard, so this is interesting.

View on HN

Yes! It first felt counterintuitive and constraining to prepare data ahead of time, rather than just loading whatever you want on the fly. But we’ve found this to be a great discipline in practice because it forces you to think about what data you actually need to show, and to shift as much of the compute to build time to reduce file sizes (through aggregation and filtering) and accelerate page load. And once you become accustomed to instant dashboards it becomes unthinkable to go back to slow queries.

Realistically what's the biggest you can make the dataset if you need to prebuild everything? Aggregation and filtering help, but quickly become impractical if you want people to change the filters dynamically.

I think the idea of Framework is really good, but static data limits the applications, excluding monitoring and other cases in which the data is constantly changing, but the dashboard can stay as it is. For example, I'd love to see a revamped Framework version of the LHC beam monitor and related pages (see https://op-webtools.web.cern.ch/vistar/, but check again in 2 months or so, when the accelerator will be running).

In high-energy physics, ROOT is /the/ toolkit for data analysis, and I guess jsROOT (https://root.cern.ch/js/) could also be used to load data to be shown in Framework dashboards. I thought the idea of Framework as a blogging engine with powerful data visualization built-in could be very interesting. Think, for example, about physicists pulling open data (https://opendata.cern.ch) and writing about their analysis or someone pulling data from https://ourworldindata.org/ in their own visualizations to support their case while writing about a particular subject, etc.

Me too, and that lead to developing the « datasette-dashboards » plugin[0]. I use this for my company where all the data is gathered by connectors scheduled in CI, storing data in Git, and triggering a SQLite db build and Datasette deployment. « BI as Code » if you will

[0] https://github.com/rclement/datasette-dashboards

How often do you refresh the data (how often your CI runs)?

Depends on the nature of the data and the requirements, can be every hour or once a day.

Most people and organizations have small/mid data and most of all slow data: you do not need fresh data every hour. If you do, you need a way heavier real time analytics data

Reply on news.ycombinator.com