Is anyone familiar with any Rust language data libraries for data engineering? My experience with Rust is it's too low level and verbose to be productive at reshaping and cleaning data.
I still check _Are we learning yet_ [3] to see their recommendation on the state of the ML stack (including data processing and structures) in Rust. It still lists "the ecosystem isn't very complete yet.", which is how I felt at the time!
[1] https://github.com/google/evcxr/tree/main/evcxr_jupyter
[2] https://datacrayon.com/shop/product/data-analysis-with-rust-...
The downside is that you have to cache data, there is a latency penalty on the first request for any new data. One would expect performance to be poor when querying parquet directly using these new lazy dataframe libraries, but it’s actually amazingly fast. Very surprising.
Thanks!
SqlAlchemy -> Sqlx or SeaORM
psycopg2 -> postgres and tokio-postgres
requests -> reqwest
BeautifulSoup -> scraper
Searching for packages on lib.rs, crates.io or github usually works.
There will be differences of course. The ecosystem is not as complete. On the other hand, writing the code and having it work the first time is easier once you know the language.