Data Engineers Spend 2 Days per Week Firefighting Bad Data

What a weird article - poor data quality impacts 26% of their companies’ revenue - how in the world do you even begin to calculate something like this? It reminds me of "X% of revenue lost to software piracy!" bs. Gotta sell those "helpful" data products I guess...

There's always going to be a lowest quartile (or whatever) quality of data. Remember the recent SRE thread? You should be modeling "data quality budgets" like "error budgets". It's also naturally corrective, if you spend less than Xd/w on firefighting you're going to start scaling up/out features or scaling down costs until you are spending Xd/w dealing with data fires again.

In the end, data engineers exist because "bad data" exists. That's the job.

I could imagine a simple use case where the revenue reported is off by 26% and bam, 26 % of revenue impacted. You could then go into revenue attribution for another revenue impact.

Yup. I know. A lot of my time is spent fixing crap data.

If not more!