The Spark community (LinkedIn) developed Dr Elephant to profile jobs and provide suggestions for reducing memory/cpu consumption. Metaflow would need something similar:
I've seen that, yeah. I've already implemented a memory profiler for Python batch jobs (https://pythonspeed.com/fil), but starting to think about how to integrate it into specific pipeline frameworks.