1. Import data using hand-written SQL from SQL server (read_sql)
2. Perform various filtering, aggregation, math (loc, iterrows/itertuples, groupby, agg)
3. Push results either out to delivery file or back to source server (to_csv, to_excel, to_sql).
Really bread and butter stuff, but the relative ease and stability of using pandas to do it is the attraction.
- two signals X and Y should always be within 5% of each other while condition Z is true.
- signal A should transition from value M to N within time window t1-t2
Would love something in rust to match the rest of the infrastructure. Python also has plenty of warts with regards to packaging, cross compilation and deployment
1) Speed and memory efficiency, I've often worked in python/pandas and been frustrated by its performance. If I have a job that takes 12 hours to churn through all of the data, flipping over to rust may accelerate my iteration time to ~7.2-72 minutes.
2) Straightforward FFI bindings into other languages. Like C you can trivially bind rust code into python. Language choice can then be a bit of a two-way door back to python.
In most of my projects I'm unable to avoid using a udf/python data manipulation at some point.