Or is the idea that you specify the type of dataset you want, and then wait a few hours for the table to be generated from the activity stream?
Good luck with your launch, this is a cool and novel take on data analytics.
One reason for this is that most warehouses are column-oriented. This means our table, which is 11 columns wide but pretty deep, is really fast to query.
We had a customer on a 3 Node (tiny) DS large Redshift warehouse dealing with billions of rows. Their Looker queries literally took 8 hours. We moved them to the Narrator structure and they saw the same queries assembled using the activity stream down to minutes (<5 minutes)
Edit: I should also point out that all queries in Narrator are compiled to SQL that runs directly on the warehouse. We don't use any external data stores or anything like that.
1. you define activities, each with an associated SQL query eg customer opened support ticket: SELECT * FROM tickets WHERE customer_id=$customer_id
2. users use Narrator UI to build a report, similar to Looker
3. Narrator creates a table for that report in the same db and populates it with data, based on 1.
4. Narrator maintains all the reports tables (updates, deletes when report is deleted)
?
2. The report (we call it Dataset) you build with the Narrator UI is a table that you can aggregate different ways, plot, and export (including writing back to the warehouse as a materialized view)
3. done optionally as part of 2
4. Yes, we keep anything written back to the warehouse up to date. You can control the cadence.
Because of 4. we work well with BI tools like Looker. Once you have the data you want just point Looker to the right table in the warehouse.