Comment by itamarst | Hacker News Reader

Comment by itamarst | Hacker News Reader

Parent Full thread

Hey, been meaning to reach out.

There's a bit in the Metaflow docs that talks about choosing resources, like RAM: "as a good measure, don't request more resources than what your workflow actually needs. On the other hand, never optimize resources prematurely."

The problem is that for memory, too little means out-of-memory crashes, so the tendency I've seen is to over-provision memory, which ends up getting very expensive at scale.

This choice between "my process crashes" and "I am incentivized to make my process organizationally expensive" isn't ideal. Do you have any ways you deal with this at Netflix, or have you seen ways other Metaflow users deal with it?

I have some ideas on how this could be made better (some combination of being able to catch OOM situations deterministically, memory profiling, and sizing RAM by input size for repeating batch jobs), based in part on some tooling I've been working on for memory profiling: https://pythonspeed.com/fil, so would love to talk about it if you're interested.

I'd love to hear more what you have in mind! Feel free to drop by at our chat at https://gitter.im/metaflow_org/community

While it is true that auto-sizing resources is hard and the easiest approach is to oversize @resources, the situation isn't as bad as it sounds:

1) In Metaflow, @resource requests are specific to a function/step, so you end up using resources only for a short while typically. It would be expensive to keep big boxes idling 24/7 but that's not necessary.

2) You can use spot instances to lower costs, sometimes dramatically.

3) It is pretty easy to see the actual resource consumption on any monitoring system, e.g. CloudWatch, so you can adjust manually if needed.

4) A core value proposition of Metaflow is to make both prototyping and production easy. While optimizing resource consumption may be important for large-scale production workloads, it is rarely the first concern when prototyping.

In practice at Netflix, we start with overprovisioning and then focus on optimizing only the workflows that mature to serious production and end up being too expensive if left unoptimized. It turns out that this is a small % of all workflows.

One way I deal with this problem in an alternative workflow manager (Nextflow), is by calculating the memory requirement for the ~95th percentile of a job, and submitting with a rule "If this crashes from going OOM, re-submit with memory*N" (up to some max number of retries/RAM). This lets most jobs sail through with a relatively low amount of RAM, and the bigger jobs end up taking a bit more time and resources.

The better your estimator function, of course, the tighter constraints you can use.

In many cases the memory usage is linear with input data, so you can come up with a function that predicts memory usage, add some padding, and then you don't need retries. E.g. this example here: https://pythonspeed.com/articles/estimating-memory-usage/

The Spark community (LinkedIn) developed Dr Elephant to profile jobs and provide suggestions for reducing memory/cpu consumption. Metaflow would need something similar:

https://github.com/linkedin/dr-elephant

I've seen that, yeah. I've already implemented a memory profiler for Python batch jobs (https://pythonspeed.com/fil), but starting to think about how to integrate it into specific pipeline frameworks.

It would be great if the infra layer can provide some help on automated resource scaling, especially for RAM. The ML solver/tooling layer has also been making progress on this front, for example Dask for limited-RAM pandas, h2o.ai has limited RAM solvers, xgboost has an external memory version, pytorch/tensorflow models are mostly trained on SGD and only needs to load data batch by batch. It's nice that Metaflow can integrate with any python code and thus benefit from all of the efforts made on the solver/tooling layer.

Reply on news.ycombinator.com