Nebyou

Definitely high up on the priority list! We've actually been experimenting with this internally.

This is not a simply a hosted airbyte instance. We use airbyte's connectors for its common standards and the active community behind them. That being said, our use of the project is both limited, customized, and deeply embedded under our app. We do not use any UI components from Airbyte.

Thanks for raising this. We share your respect for the intellectual property rights of others. We are aware of Airbyte's license structure and use connectors per its terms.

We've observed that our system performs really well handling most corner cases as long as the context required can be interpolated from its inputs (either in the schemas and their descriptions or in the underlying data we sample from). In the worst case, the most you'd have to do is edit schema descriptions on our platform to include the necessary context (For example, specifying the encoding that you expect the field in your end schema to have).

For the compound transform scenario, since we optimize for modularity in the transformations we build, our systems prioritize defining these transformations unless it makes no sense to do so.

Apologies, I misread your earlier question. Lume can only ascertain information from what is given directly or from what assumptions it can make with reasonable confidence. So, in this case, this will not be possible unless there is any information in the start schema where this can be interpolated. If these prior transformations were done in a dbt project, we could extract the information needed for this easily (a dbt project can be integrated and represented as a separate flow within your workspace).

This example is a full refresh, but incremental is usually the norm, especially for our supported connectors and continuous syncs. Our models can detect incremental columns.

Audit timestamps (usually tables) are typically created in intermediary stages (whose materializations you would have access to in your database) before getting pruned out to fit your destination schema. Of course, if the destination schema expects these audit tables or columns, they would be included in the target.

To your last question, if you include these tables or columns in the end schema you specify to Lume (or create a separate flow with a new end schema with these fields), what you described is definitely possible.

It's true that our platform can be used for the same use cases as some of those products. However, the main difference is in the customizability we offer. These products focus on and support the most common integrations and offer them as an automation service. For most custom integrations, users still have to write custom code within these products if possible, or build them out in-house. With Lume, this would not be necessary.

Our transformations are executed in a staging database/schema before deployment. We also have versioning and backtesting capabilities. In addition, you will have complete visibility of the code we produce before and after deployment.