Artem, the approach you guys are using is really EXCELLENT!

I think that a bit of a disconnect here may be because some OPs might be used to 'compiling' code versus 'compiling' data angle that you are using.

This is especially evident by make dependencies discussion with lars512.

To give a simple specific example: I have a dataset of say 5000-50000 SKUs that are aggregated across 9-12 dimensions. My final report/analysis uses 3 scenarios. Now one sub-set of one scenario has changed [that's the raw input] - of course running 'data compilation' by using data that changed and ONLY what depends on it is the most effective&efficient approach.

Just my 2 financial cents...