westandskif

supports aggregations, joins + now optional collection items

Just to add my previous answer: the trivial native python equivalent doesn't have the same functionality, because it consumes data iterator 3 times in your case, while convtools would consume it only once.

Re: whether it's been a problem in my experience -- sort of - yes

So now I'm doing my best to observe: PEP-20 the 2nd commandment with the hope that I'm not violating the 1st commandment badly :) https://www.python.org/dev/peps/pep-0020/

Also I see another upside of this no-magic syntax in that it is distinctive -- there's no way to mix up convtools-related code with any other python code.

In terms of the implementation, it's kind of trivial to implement the "magic", but it would be both confusing and inconsistent, see below.

e.g. imagine a case where you'd want to call datetime.strptime, partially initializing it at the moment of conversion definition. at the moment it is:

  c.call_func(
      datetime.strptime,
      c.item("updated"),
      "%Y-%m-%d"
  )

but it's unclear to me how would the "magic" approach deal with the case above.

We would need to bring the whole python into it :/

Pandas are great, but I had a few cases where I had frustrating experience -- dealing with Decimal & float columns is a pain (missing data without any signs when using both in calculations).

However this was not the reason why I needed to build convtools, I needed to process reports, touching only some columns (without failing if an unrelated column is no longer processable). So I needed to reuse and combine python expressions across multiple procedures.

There are no benchmarks at the moment, you can just pass debug=True to the gen_converter method to see the generated code and judge whether it's optimal for your use case. This is a python library which generates simple python code: - without unnecessary conditions and loops - without keeping all items of iterable in memory to aggregate (it leverages reducers) - making no use of C-extensions.

It's not possible to skip the code generation part because a resulting converter is always compiled from the code written under the hood. Could you please share what is your concern about this? I'd really like to better understand it!

JFYI: it's possible to skip running "gen_converter" method, it's possible to just use "execute" -- it runs "gen_converter" under the hood: c.group_by( c.item(0) ).aggregate({ c.item(0): c.reduce(c.ReduceFuncs.Sum, c.item(1)) }).execute([ (0, 1), (0, 2), (0, 3), (1, 10), (1, 12), ])

  Out[5]: [{0: 6}, {1: 22}]

The downside is that you won't be reusing the converter.

On exception it populates linecache so that tracebacks are normal and you can debug it normally with pdb post-mortem debugging - https://docs.python.org/3/library/pdb.html#pdb.post_mortem

Nothing else at the moment, but I've written down this point to contemplate in the nearest future -- thank you!

Thank you! I will add join and group_by examples shortly

oh, thanks, I'll add links to cheatsheet and quickstart pages to the README, it really makes sense!

As for the magic-stuff, I was contemplating designing the API with this approach, but I changed my mind because it would be difficult to tell which python expressions are evaluated at the moment of a conversion definition AND which in the compiled code.

However if we imagine this "magic" API, then it could be even closer to normal python code:

  m["key"].some_method(...)

which would resolve everything under the hood.

===

as for the collapsable generated code examples -- I've jotted down :)

thank you very much for the feedback! this is very valuable! :)

Regarding the README, I'll improve it within the next few days.

As for the approach, the main assumption was that everything is simple as long as you deal with expressions only, so I've introduced every expression I needed as a conversion object (each able to generate the code within the context).

Exceptions are custom code generating parts (e.g. aggregate, reducers) and the part where I break down piped conversions into a series of statements in the top level converter.

Another tricky piece was to support parametrization - e.g. c.input_arg here - https://convtools.readthedocs.io/en/latest/cheatsheet.html#c... So it was necessary to make every conversion know about every inner dependency it has, to make all dependencies pop up, to know function signatures & parameters needed to be passed during internal generation of functions.