Hacker News Reader

Comment by gregw2 | Hacker News Reader

The demo shows you transforming As-Is data to As-Is data.

Is this a full refresh each time or incremental (do you have to tell it the incremental columns or can it "tell"?)

Can you create audit timestamps in the target which track when rows were inserted or updated (or soft-deleted) in the target?

Can you take sources which contain "current state" table information and transform them into tables that have record start-effective/end-effective date (+ a current record indicator flag?) that support As-Was querying for a given primary key and/or which tracks soft deletes over time as an additional target table column?

View on HN

This example is a full refresh, but incremental is usually the norm, especially for our supported connectors and continuous syncs. Our models can detect incremental columns.

Audit timestamps (usually tables) are typically created in intermediary stages (whose materializations you would have access to in your database) before getting pruned out to fit your destination schema. Of course, if the destination schema expects these audit tables or columns, they would be included in the target.

To your last question, if you include these tables or columns in the end schema you specify to Lume (or create a separate flow with a new end schema with these fields), what you described is definitely possible.

How does Lume know what to fill in on the target table if a record effective end date isn’t in the source system but is a property of when the data was fetched? How does Lume know to update the record effective termination date of a target row when a new row comes along?

(This type of data modeling is described here: https://www.kimballgroup.com/data-warehouse-business-intelli... )

Apologies, I misread your earlier question. Lume can only ascertain information from what is given directly or from what assumptions it can make with reasonable confidence. So, in this case, this will not be possible unless there is any information in the start schema where this can be interpolated. If these prior transformations were done in a dbt project, we could extract the information needed for this easily (a dbt project can be integrated and represented as a separate flow within your workspace).

Reply on news.ycombinator.com