Hacker News Reader

Comment by red_hare | Hacker News Reader

I will never understand python API developers obsession with overloading the operator magic methods.

Apache Beam is one of the worst offenders to me. The recommended method is to connect your functional transforms when building your pipeline as if it were a series Bash commands piped together (overloading bitwise OR). Who thought “oh, you know what language and API is synonymous with ‘write a large complicated amount of business logic in me’, Bash.”

View on HN

Your comment made me smile because Apache Beam is one of my favorite tools, but it seems nobody outside Spotify has had heard of it.

The bitwise operation dunder methods have no definition outside of integers. So they’re basically freebies to do what you want with.

Honestly I think the Apache Beam API makes a lot of sense.

This is because it started out as a Google project. There is an internal time-series framework there called Monarch that does the same thing for composing operations.

I have to agree with this sentiment.

To be clear, I love Beam. It runs all of my startups stream and batch processing.

I just think the Java API's version...

    pipeline = pipeline
      .apply("FirstTransform, MyFirstParDo())
      .apply("SecondTransform, MySecondParDo())

is much clearer than the Python version...

    pipeline = (
        pipeline
        | "FirstTransform" >> MyFirstParDo()
        | "SecondTransform" >> MySecondParDo()
    )

Agreed. I prefer the Java version, for a number of reasons. Scio is even better. But the bitwise operations make sense python wise.

Oh man, I ran into that when I made my first contribution to our internal Airflow project and was just like wtf is happening.

To be fair it's pretty nice when you get used to it as I doubt you're going to be using bit shifts much.

It's fine. But there are some unintuitive edge cases.

Like, for example, if you have tasks A, B, C, and D you can do...

    A >> (B, C) >> D

but not...

    (B, C) >> A >> D

you have to write...

    D << A << (B, C)

Reply on news.ycombinator.com