> With Hamilton we use a new paradigm in Python (well not quite “new” as pytest fixtures use this approach) for defining model pipelines. Users write declarative functions instead of writing procedural code. For example, rather than writing the following pandas code
> These functions then define a "dataflow" or a directed acyclic graph (DAG), i.e. we can create a “graph” with nodes: col_a, col_b, col_c, and col_d, and connect them with edges to know the order in which to call the functions to compute any result.
This 'new paradigm' already exists in Polars. Within the scope of a local machine, you can write declarative expressions which can then be used pretty much anywhere for querying instead of the usual arrays and series (arguments to filter/apply/groupby/agg/select etc), allowing it to build an execution graph for each query, optimise it and parallelise it, and try to only run through the data once if possible without cloning. Eg the example above can be written simply as
col_c = (pl.col('a') + pl.col('b')).alias('c')