Databot: High-performance Python data-driven programming framework | Hacker News Reader

Databot: High-performance Python data-driven programming framework

github.com

94 pointsgarygoog22 comments

I was staring at the example in the readme, and considering some of the features ("replayable"), and it sounds a bit like what a Makefile does. Well, if you had a single event to process, and decided to use the filesystem to store input, intermediate results, and output.

So here's an implementation of the example from the readme using make, and bash, and jq, and a silly python script to implement a timer by modifying a file every X seconds:

  ~/projects/makething$ cat Makefile 

  default: d

  b:  a
      curl -o $@ "http://api.coindesk.com/v1/bpi/currentprice.json"

  c:  b
      jq '.bpi.USD.rate_float' $< > $@

  d:  c
      cat $<
      cp $< $@


  ~/projects/makething$ cat timer.py 

  import sys
  import time

  def main():
      delay = float(sys.argv[1])
      fn = sys.argv[2]
      while True:
          with open(fn, 'w') as f:
              f.write(str(time.time()))
          time.sleep(delay)

  if __name__ == '__main__':
      main()


  ~/projects/makething$ cat go.sh 

  #! /usr/bin/env bash

  python timer.py 2.0 a &
  TIMER_PID=$!

  function cleanup() {
      kill $TIMER_PID
  }
  trap cleanup EXIT
  
  while true
  do
      while make -j -q
      do
          sleep 0.1
      done
      make -j
  done

make decides if things are up-to-date by comparing timestamps of files in the filesystem, so we can emulate a timer that triggers an event every 2 seconds by having a process modify a file every 2 seconds, and rig a rule in our makefile to use that file as an input.

The interpreter pattern permeates the stack.

I am a huge fan of using `make` for this sort of ad hoc data pipeline. The workflow is very natural, as you can play around with each step on the command line and then drop it into the makefile once you get it right..better for reproducibility than search up through terminal history to replay individual lines!

In your example, I would drop the shell and python scripts and simply run:

    watch -n 2 -d "touch a && make"

thank you for the review and the improvement! much cleaner.

from an abstraction point of view, there is no penalty for wrapping this in some better-for-humans layers above it.. dispatch - with a bit of discipline - is almost costless.

<opinion> for decades the *nix culture suffered from "suffer or you arent smart" syndrome, in my opinion. There is nothing wrong with easy, or nice to look at, its just that it is rare to find quality in both the implementation layer (like this) and the interface choices, and then to keep it simple-stupid going forward. I like the simplicity of this, btw</opinion>

Recently I started to use Apache NiFi[1] for everything that does not required too complex operations. It's pretty much what this framework does, just with an UI and a lot of monitoring features.

However, one downside is the massive RAM consumption. 1GB of RAM even if it does pretty much nothing is quite a bill to start off with.

1: https://nifi.apache.org/

Yes, but Nifi is pretty good at what it does. 1GB RAM cost is nothing compared to the time it saves you in the end to make very robust data flows.

Of course 1GB RAM is nothing nowadays, but this was something that I wanted to point out since sometimes you have constrained resources available or need to calculate what machine size you want to use.

NiFi might not run on an AWS t2.micro instance. Whereas Apache Airflow does.

When you use Nifi usually you are not in a budget constrained environment. Because CPU wise it takes its toll too and you need more than one core to be comfy, so micro instances are out anyway.

I would rather use the 1 GB closer to the problem I want to solve.

So basically Apache has two separate projects doing the same thing -- NiFi and Airflow? Or are they significantly different?

They are, but the significance depends on your view point. Both provide some sort of "ETL capability". For me the deciding factors were if I wanted to have bulk operations over many records or basically a state machine for individual records. NiFi works great on the latter, as far as my experience goes, Airflow for "get whole data file and process it as a whole and then move to the next processing step".

EDIT: plus, of course, things like your tech landscape. NiFi is Java and Airflow Python. If all of your tools are around one or the other, you can get both to work for the same use cases, but as mentioned above you will have more or less implementation work to do.

>So basically Apache has two separate projects doing the same thing

As someone else said Airflow and NiFi are decently different. However, Apache having multiple projects that overlap is pretty common.

Streaming you have Flink, Spark and Storm. Batch processing you have Flink, Spark, Hadoop MapReduce, Hive, and Tez. Serialization you have Avro and Thrift. Columnar storage you have ORC and Parquet. Batch ETL management you have Oozie and Airflow. NoSQL you have Cassandra and HBase. That's just off the top of my head.

https://www.enterpriseintegrationpatterns.com/patterns/messa...

https://www.coursera.org/lecture/software-architecture/3-2-7...

Another project relying on lamdas for data processing https://github.com/asavinov/lambdo yet focused more on feature engineering and ML

This reminds me a lot of reactive programming like ReactiveX [http://reactivex.io] which has a python implementation

> which has a python implementation

I think this is the most popular one:

https://github.com/ReactiveX/RxPY

This one is a rewrite of RxPY, that makes use of async / await / asyncio:

https://github.com/dbrattli/aioreactive

Pretty interesting stuff!

Somehow I would expect pipes to be connectable and nestable. This does not seem to be the case (by studying the source). Then you could have some function to build and parametrise a sub-pipeline and connect them to something bigger.

I'm still looking for a perfect natural python ETL dsl, so I will follow that project.

So far I'm using https://github.com/petl-developers/petl and mostly happy with it.

> I would expect pipes to be connectable and nestable.

How would that look like? I mean a pipe is something where one writes data into and another loads data from in the same order it was written into. I don't see how that could be nested, or why two pipes would need to be connected.

This python project is modelled on yahoo pipes is worth taking a look at: https://github.com/nerevu/riko

So it's like Nifi, but not as good? What would be the benefit to use that?

Isn't using "performance" and "python" in the same sentence an oxymoron?

Reply on news.ycombinator.com