dancrystalbeach

LLMs are not that scary. You can easily run one on your laptop. Pull back the curtain, no magic.

Python vs Rust, Data Engineering Conundrums

Recursive CTE's in SQL. Devil or Angel?

Will Rust really take over the (data) world?

MLOps basics and diving into Model Training.

Machine Learning Basics.

Dealing with Stress and Anxiety for programmers.

What's it like processing Billions of records with Databricks + Delta Lake. Find out.

The two hottest tools in Data Engineering right now, DuckDB and Polars. Who will win?

Mixing native Python with PySpark?

DSA for the rest of us. QuickSort with Rust.

Simple introduction to Binary Search in Rust.

Rust vs Python is always an interesting take. They are such different languages.

Data Engineers suck at unit testing. They shouldn't.

The classic problem of ambiguity in Software Engineering.

Looks like everyone is trying to save money. Instead of laying people off, I think companies should inspect their cloud architecture and processes.

Long Live Apache Airflow.

SQL has engulfed the data world. Can we ever break free?

I've thought about that, i should probably do more detailed analysis of what the actual cost difference would be. I've looked at Presto as well... but again that is just another tech stack to manage.

the consumption is very SQLish based. "operational toil is higher with Spark" <- I agree with this, this is why I'm leaning towards Redshift. Throwing away half the codebase in favor of some SQL statements is tempting.

Very interesting. Those ML models must be huge, I wonder if they are using Spark or something else to run those models.

It's seems like there is an ongoing battle in Python to adopt... or not... Aysnc. It's complicated to implement, causing slow and low adoption.

Metadata management is becoming a big deal in data engineering these days. It's hard to find good metadata management that doesn't tie you directly into some product or ETL tool. We need more agnostic metadata management tools.