Show HN: Pipelines – Language for scripting parrallel pipelines with Python | Hacker News Reader

Show HN: Pipelines – Language for scripting parrallel pipelines with Python

github.com

69 pointscalebwin16 comments

This looks pretty interesting.

Just curious if you looked at Apache Airflow as the integration functionality and pipeline / DAG configuration approaches are similar and it's also done in Python.

https://github.com/apache/airflow

The Nim language is new to me. What was the experience like implementing a framework in it?

Looks interesting but I don’t understand if the “x as x” is mandatory? Seems overly verbose if you don’t want to import it with another name?

That is unnecessary, I don't understand why the Readme includes it. It does nothing.

It's a relatively simple Nim code. It takes a line that starts with "import" and searches for three components: module, function and alias. If you don't provide an alias, you'll get an index error.

I'm not sure I understand the wins of this over just using the standard multiprocessing module.

You can trivially define a worker pool or arbitrary size and pipe a stream of values through your function using multiple cores, avoiding GIL issues.

If you have something which scales to multiple machines just use celery or redis streams.

DASK

Nice readme! It's so common for projects to forget to include an introductory blurb, it's great to see one that does not forget.

Small bug in the example code? I think this:

    return even % 2 == 0

Should be this:

    return number % 2 == 0

I didn't understand one thing: is this implemented in Nim and run Python code? If yes, why don't run Pipelines in Nim too? Wouldn't this be faster?

Most programmers don't write in Nim. More programmers write in Python. I assume he's targeting the Python crowd.

wait... `/>` and `|>` are python operators?

https://docs.python.org/3/reference/datamodel.html#special-m...

No. Those are Nim, or rather tokens that guides the Nim code to do the pipe thing.

The code splits the line in function names, args and those operators, to then orchestrate the pipeline flow.

Should have named it Pypelines. But seriously though I am not sure what the benefits over multiprocessing are. It looks a little cryptic at first sight.

https://github.com/gpcimino/pypelines

Looks very similar to R, dplyr and pipes.

More like magritter I thought. just wish everytime someone comes up with something new you have to install yet another package manager.

Reply on news.ycombinator.com