Data Warehouse Refactor Approach
As part of migration from Postgres to Snowflake, we are planning to refactor our ETL pipeline.
I have suggested below 2 approach
(1) Document existing database and pipeline dependency and design new solution
(2) Migrate pipeline as it is. As refactor later bit by bit.
But leadership wants a hybrid approach i.e start with development and refactor on the go. As per my experience in working in many migration project this is a recipe for disaster.
They don't want approach (1) because this will delay delay the project and nothing to showcase to business.
They don't want approach (2) because once migration is complete, business won't see any benefit in refactoring.
In my experience working in industry for last 11 years, migration and refactoring with time limit never works and solution ends up been a disaster.
So far, I have worked in IT services and this is the first time, i am working on project for own company.
I am thinking too much about the outcome and failure ?