Also I'm happy to answer any questions (I lead the Metaflow team at Netflix).
There's a bit in the Metaflow docs that talks about choosing resources, like RAM: "as a good measure, don't request more resources than what your workflow actually needs. On the other hand, never optimize resources prematurely."
The problem is that for memory, too little means out-of-memory crashes, so the tendency I've seen is to over-provision memory, which ends up getting very expensive at scale.
This choice between "my process crashes" and "I am incentivized to make my process organizationally expensive" isn't ideal. Do you have any ways you deal with this at Netflix, or have you seen ways other Metaflow users deal with it?
I have some ideas on how this could be made better (some combination of being able to catch OOM situations deterministically, memory profiling, and sizing RAM by input size for repeating batch jobs), based in part on some tooling I've been working on for memory profiling: https://pythonspeed.com/fil, so would love to talk about it if you're interested.
While it is true that auto-sizing resources is hard and the easiest approach is to oversize @resources, the situation isn't as bad as it sounds:
1) In Metaflow, @resource requests are specific to a function/step, so you end up using resources only for a short while typically. It would be expensive to keep big boxes idling 24/7 but that's not necessary.
2) You can use spot instances to lower costs, sometimes dramatically.
3) It is pretty easy to see the actual resource consumption on any monitoring system, e.g. CloudWatch, so you can adjust manually if needed.
4) A core value proposition of Metaflow is to make both prototyping and production easy. While optimizing resource consumption may be important for large-scale production workloads, it is rarely the first concern when prototyping.
In practice at Netflix, we start with overprovisioning and then focus on optimizing only the workflows that mature to serious production and end up being too expensive if left unoptimized. It turns out that this is a small % of all workflows.
The better your estimator function, of course, the tighter constraints you can use.
Metaflow is rather unopinionated about those types of questions, since they are subject to active research and experimentation. Metaflow aims to make it easy to conduct the research and experiments but it is up to the data scientist to choose the right modeling approach, features etc.
In some cases, individual teams have built a thin layer of tooling on top of Metaflow to support specific problems they care about. I could imagine such a layer for specific instances of transfer learning, for instance.
In general, we are actively thinking if/how Metaflow could support feature sharing in general. It is a tough nut to crack.
Or you're forever tied to the initial hardware+compiler version?
Depending on the libraries you use, the exact results may or may not be reproducible on other architectures. If cross-platform reproducibility is important to you, you should choose your libraries accordingly. Metaflow provides the tools for choosing the level of reproducibility that your application requires.
Also, what tools does metaflow offers to control the level of reproducibility?