Comment by swyx | Hacker News Reader

Comment by swyx | Hacker News Reader

the way I think about upskilling is it is essentially a high stakes bet on megatrends in the industry - you want to be early + right as that is how you get paid the most.

take your pick from https://airbyte.com/blog/becoming-a-data-engineer-2023 :

- Language: SQL and Python Continue to Dominate, With Rust Looking Promising

- Abstraction: Custom ETL Is Replaced by Data Connectors

- Latency: Better Tools Available In The Streaming Domain

- Architecture: The Lakehouse and the Semantic Layer Combine Strengths

- Trust: Data Quality and Observability Become an Essential

- Usability: The New Focus Is on Data Products and Data Contracts

- Openness: When It Comes to Data Tools, Open Source Is the Answer

- Standards: Database Open Standard Simplified by DuckDB

- Roles: Data Practitioners Expand Their Knowledge into Data Engineering

- Collaboration: DataOps Reduces Data Silos and Improves Teamwork

- Adoption: Data Engineering Is Key Regardless of Industry or Business Size

- Foundations: The Data Engineering Lifecycle is Evergreen

(not my article but i work at airbyte, this was the result of our 2023 trends review)

Is anyone familiar with any Rust language data libraries for data engineering? My experience with Rust is it's too low level and verbose to be productive at reshaping and cleaning data.

We use rust for data processing, etl and to serve apis directly from massive parquet collections using Polars. What would you like to know?

What are your requirements, and how do you store and load the data? Are you doing anything to select only the relevant parts of the parquet files?

Requirements largely require selecting from and aggregating across some subset of the whole universe of data, which is on the order of hundreds of GB of parquet. We use partitioning across files (and folder structure in s3) to lazily scan only the bits of data we need, then predicate and projection push down (thanks to polars) to reduce that further.

Thanks. If you don't mind, what's the size of your individual files, and how many of them does it take to serve a single request: one, a few, N?

Variable, but a single partitioned file goes anywhere from single-digit kb to maybe a little under a meg. A query can touch just one or dozens (all lazily concatenated into a single frame).

Why serve APIs using parquet collections as a data source, as compared to a standard RDBMS? What are the pro's and con's?

We’re talking about hundreds of gigabytes, pushing towards many terabytes of data with extremely sparse access patterns. Loading into anything (including an rdbms) wouldn’t be cost-effective or practical.

The big pro is that the only data that ever has to move is that which is actually accessed. It’s extremely cost-effective to produce, store and query statically vs moving it into something else of sufficient scale to handle the entire universe of data.

The downside is that you have to cache data, there is a latency penalty on the first request for any new data. One would expect performance to be poor when querying parquet directly using these new lazy dataframe libraries, but it’s actually amazingly fast. Very surprising.

Using Parquet files is more scalable because the amount of data can be vast. It can be impractical to use a normal RDBMS to hold billions of rows, but is no problem for a bunch of files sitting in S3 or file storage. It can also be radically cheaper because you're just paying for file storage. The Parquet file format is also designed to be both compressed but also skippable to find the rows you're looking for. Also, Parquet files can be mounted in a filesystem for usage with things like Spark.

I'm a Python dev, learning Rust out of interest, but also considering its viability for Data Engineering. It would be helpful to know the equivalents of the following libraries: Pandas (Polars), Prefect/Airflow/Orchestration, Flask, SqlAlchemy, psycopg2, requests, BeautifulSoup.

Thanks!

Flask -> Axum (see also https://www.arewewebyet.org/topics/frameworks/#server)

SqlAlchemy -> Sqlx or SeaORM

psycopg2 -> postgres and tokio-postgres

requests -> reqwest

BeautifulSoup -> scraper

Searching for packages on lib.rs, crates.io or github usually works.

There will be differences of course. The ecosystem is not as complete. On the other hand, writing the code and having it work the first time is easier once you know the language.

Brilliant, I'll be playing with these once I reach the more advanced material in The Book. Ty.

A couple of years ago when checking out Rust for scientific work I started looking for familiar ground, e.g. Jupyter notebook support. I was happy to find google/evcxr's Jupyter kernel [1] and started writing an early access book on Rust for data analysis [2].

I still check _Are we learning yet_ [3] to see their recommendation on the state of the ML stack (including data processing and structures) in Rust. It still lists "the ecosystem isn't very complete yet.", which is how I felt at the time!

[1] https://github.com/google/evcxr/tree/main/evcxr_jupyter

[2] https://datacrayon.com/shop/product/data-analysis-with-rust-...

[3] https://www.arewelearningyet.com

Isn't https://arrow.apache.org/ballista/ supposed to be a kind of next generation Spark written in Rust?

Datafusion is okay. Ballista is not at all ready at the moment.

I don't know a whole lot about Rust. I think Rust is a great language for writing Database components and systems. For example plugin for PostgreSQL. Previously this would be done in C. You could still do this in C today too. Data Engineers are heavy users of databases and different data stores, but they normally don't design database systems.

Reply on news.ycombinator.com