> pipeline = task(get_data, branch=True) \
> | task(step1, workers=20) \
> | task(step2, workers=20) \
> | task(step3, workers=20, multiprocess=True)
you could reassign every line, but it would look nicer with chained functions.
pipeline = task(get_data, branch=True)
pipeline = pipeline | task(step1, workers=20)
pipeline = pipeline | task(step2, workers=20)
pipeline = pipeline | task(step3, workers=20, multiprocess=True)
I would be tempted to do something like this:
steps = [task(step1, workers=20),
task(step2, workers=20),
task(step3, workers=20, multiprocess=True)]
pipeline = task(get_data, branch=True)
for step in steps:
pipeline = pipeline.__or__(step) pipeline = task(get_data, branch=True).pipe(
task(step1, workers=20)).pipe(
task(step2, workers=20)).pipe(
task(step3, workers=20, multiprocess=True))
pipeline = task(...)
pipeline |= task(...)
steps = [task(...), task(...)]
pipeline = functools.reduce(operator.or_, steps)
pipeline = pyper.Pipeline([Task(...), Task(...)])I've not been doing Python day-to-day so I'm starting to lose my touch on all the nice little tricks.
pipeline = (
task(get_data, branch=True)
| task(step1, workers=20)
| task(step2, workers=20)
| task(step3, workers=20, multiprocess=True)
Square brackets would create a list and braces would create a set of course. The contents still can be split over different lines-- just pointing that this syntax doesn't do the same thing.