Well, you know I simply can’t resist throwing my hat in the ring. If everyone else is going to stand on a soapbox and decry the end of all data pipelines as we know it, you can be sure I’m going to climb to the top of the old oak tree and scream my truth to the glassey eye’d masses who’ve been sitting in their echo chambers for far too long.
I can see all you old-school SQL Server buggers just squirming in your seats, hot under the collar. How dare they speak of such sacred things, holy, without reference and a nod to the past? Methinks someone, somewhere, once said that there is nothing new under the sun. This indeed is true.
So, you want to join the cool kids on the block and don't know where to start? Do you have stars in your eyes and think adopting Databricks as a Data Platform is the cure-all for the data woes and problems you have today?
In this episode, I sit down with industry veteran Robin Moffatt — Sr. Principal Advisor in Streaming Data Technologies (Kafka, etc.) and a longtime voice in the data engineering community, to unpack the journey from old-school data architectures to today’s real-time streaming ecosystems. From early mainframe data processing and COBOL through the rise of Apache Kafka, streaming ETL, and event-driven systems, Robin shares lived experience from across decades of building, scaling, and evolving data platforms.
I rarely get excited about happenings in the data world. When you’ve been staring at CSV files for 20 years, heard the same problems argued over and over again with only a different tool name at the front of the sentence, things can get a little … mundane.
Andy’s career didn’t start in software at all. It started with physical circuits, literally wiring systems as an electrician, before moving into programming, databases, and eventually decades of hands-on data engineering work.
It’s an interesting time to be in software and data; the world of generative AI is changing the landscape beneath our feet. I don’t see this as a bad thing for software folk, but as an opportunity to learn new technologies and BUILD / UNDERSTAND the technologies used in an LLM and AI context.
You can’t expect an LLM trained two years ago to be up-to-date on what the new and best approaches are for X, Y, Z tech.
Every day I go out and do the Lord’s work for y’all. On an average evening or weekend, you can find me setting up cloud instances, generating data, deploying code, and running tests. I don’t answer to anyone. I write what I please and what I find. The simple approach: poke things, turn over rocks, ruffle the feathers of many powerful people.
Anyone who’s been around for more than a decade or so in the programming, development, and data world might get a slight eye twitch when the word database driver appears. Before the modern times we live in came along, the entire data world was driven by SQL Server, Oracle, with just a sprinkling of Postgres and MySQL … a-la AWS. That’s just the way it was.
It seems we have several cadres of people when it comes to “clean code.” I know there is a lot of previous baggage that comes with that nomenclature, good and bad. But, I think we can think about “clean code” from a simplistic point of view. It doesn’t have to be that complex.
Well, don’t pretend like y’all didn’t see it coming. If I close my eyes and listen to the winter wind blowing through the maple trees, I can see Josue Bogran dancing in the moonlight with joy. There is truth in what he says, “All roads lead to Excel.”
I give the people what they want; I’m a slave to clicks and the buzz of internet anger and mayhem. Not so long ago, I tried, with limited success, to convince a legion of engineers who were raised on the Snowflake and Databricks teat, that salvation lay right at their feet; all they must do is lean down and put their hand to the plow.
I’ve been a Polars bro for most of the last few years. Why? It’s Rust-based, fast, DataFrame-centric, just the way I like it. It also had the excellent feature, right from the start, of Lazy Execution. A few years ago, maybe two, I actually put Polars into production, running on Airflow, working with S3 and reading Delta Lake tables.
I am sort of an addict for a good Lake House; it’s just an incredible architecture that is a joy to build and use daily. No more SQL Server or Oracle, you know what I’m saying if you were born in the right decade.
It's a fast-paced and ever-changing world we live in; nothing we can do about it. I grew up in the middle of the prairie, when the internet became mainstream, the age of Doom, Myst, MSN Messenger, Yahoo Pool, and that irreplaceable Goldeneye, let's be honest, World of Warcraft on a PC was game-changing. I suppose you could chalk up half my feelings as nostalgia and old-person hum-drum, I won't deny it.
Can you, should you, and what will happen, if we use LLMs not just to puke out the next code snippet … but what if we use an LLM mixed with a little Agentic AI, to actually to BE the data pipeline????
I’m getting to the point where I’m a little burnt out on hearing about AI this and AI that. Work, LinkedIn, YouTube, Substack … it’s a never-ending glut of AI-generated content talking about AI. I get the sense that the closest most developers get to AI is asking Cusor to puke out the next code chunk.
Recently, I had to migrate a few hundred Delta Lake tables that were partitioned over to Liquid Clustering. It seems straightforward on the surface, but everything usually does, until it isn’t. There was no rocket science involved here, but I did want to write this up to help the myriad of others who will probably want/need to move from partitioning to liquid clustering over the next few years.
Meh. That's how I feel about the languishing and short-lived ideology called “Data Mesh.” The Modern Data Stack has seen its fair share of prophets seeking the loving embrace of the community, a fickle thing, only to fall out of favor all too soon.
Oh, that age-old topic of Data Quality, often discussed but never implemented. Poor little blighter. Who knows what it’s like to be the old, beat-up rock that always gets kicked, gets no love or attention, just told about everything you do wrong
Not only is it not cheap to crunch 650 GB of data on a Spark cluster —piling up DBUs, truth be told — but it’s not complicated either; they’ve made it easy to spend money. Especially when you simply don’t need a cluster anymore for *most datasets and workloads.
I am certainly no master of the minutiae of Databricks compute, able to unwind the wound-up ball of string that leads us to where we are today. Yet DBUs matter in the world of the Lake House. It’s the difference between an angry CTO and a happy one.
in an effort to bring value and push the boundaries of what is possible, we must at times stray from the safe path. Of course, the minute we stray from the well trodden way, we can expect trouble around every corner.
We lived in a world where we hear the words ... "Terabyte", or "Iceberg" or "Lake House" and we think Cluster. We ... must ... have ... Cluster ... give ... me ... cluster. DBUs start piling up. We smile, our boss smiles, Databricks smiles. Just a little pipeline, another one of a million pulsing through the network today. What's just one more?
Things have changed a lot in the last year related to LLMs and AI; on the one hand, it seems the AI skeptics for coding are increasingly confined to the corners of the internet. Everyone is dancing around in the middle, not sure of where everything should fall.
The real question is, do you think Spark is the bloody British, the AWS EC2 instance is Boston, and DuckDB + Polars are the John Adams and George Washington, bent on freedom or death? You paid attention in history class, didn’t you?
I was recently working on a PySpark pipeline in which I was using the JDBC option to write about 22 million records from a Spark DataFrame into a Postgres RDS database. Hey, why not use the built in method provided by Spark, how bad could it be? I mean it’s not like the creators and maintainers of Spark aren’t probably our version of rocket engineers.
I can no longer hold the boiling and frothing mess of righteous anger that starts to rumble up from within me when I hear the words “Medallion Architecture” in the context of Data Modeling, especially when it’s used by some young Engineer who doesn’t know any better. Poor saps who have been born into a Databricks world where that fresh, supple mind has been polluted and twisted by the machinations of a marketing department.