I don’t know about you, but I grew up and cut my teeth in what feels like a special and Golden age of software engineering that is now relegated to the history books, a true onetime Renaissance of coding that was beautiful, bright, full of laughter and wonder, a time which has passed and will never return.
I’m such a cheerful guy, I figured I would try to write a positive and inspiring post while comparing two tools. Just kidding. I guess we will see how it goes. Who knows, we'll just see where it leads us, I suppose.
You know, after literally multiple decades in the data space, writing code and SQL, at some point along that arduous journey, one might think this problem would be solved by me, or the tooling … yet alas, not to be.
I’m going to give you the spoiler up front. AI isn’t taking your job anytime soon. If you think otherwise, you’ve been watching too much YouTube and reading the wrong stuff on LinkedIn. Heck, maybe you simply HAVEN’T been using AI enough on more than junior-level problems.
ometimes, I force my old caveman self to do things I do not want. Believe it or not, reviewing the landscape as it relates to data orchestration frameworks is not something on the top of my list. People, including me, have their own pet favorites when it comes to orchestration platforms, and it’s hard to be unbiased.
When it comes to the AI hype, I pretty much have tried to ignore the constant roll of never ending models and other hoopla that is mostly meaningless. Better to let the dust settle.
I simply can’t resist the urge to put my ring in the hat on this one. It's mostly because I see the glut of AI nonsense put out by talking heads who’ve never written a single line of usable code in their lives.
But what’s also intriguing is the takes by well-seasoned Software Engineers who’ve been writing code for decades and who are apparently taking different sides in this battle. What battle, you ask? The computers vs humans battles.
I’ve never been one who’s much for self-help, you know, all those books you walk past in the airport or at Barnes and Noble. “10 Ways to Transform Yourself …” Yeah, right, I usually snort to myself and walk on.
When it comes to building modern Lake House architecture, we often get stuck in the past, doing the same old things time after time. We are human; we are lemmings; it’s just the trap we fall into. Usually, that pit we fall into is called Spark. Now, don’t get me wrong; I love Spark. We couldn’t have what we have today in terms of Data Platforms if it wasn’t for Apache Spark.
You know, for all the hoards of content, books, and videos produced in the “Data Space” over the last few years, famous or others, it seems I find there are volumes of information on the pieces and parts of working in Data. It could be Data Quality, Data Modeling, Data Pipelines, Data Storage, Compute, and the list goes on. I found this to be a problem as I was growing in my “Data” career over the decades.
Sometimes I find myself lying in my sunroom, staring out the window in the blue sky above me while the sun plays on the maple tree, empty of most all but a few red leaves … wondering what else I can do to make the already angry readers of my babbling even more angry.
Maybe it’s my age, I’m not totally sure, but it’s getting harder and harder for this old guy to keep up with all the new things that come trickling out of the quagmire that is the Data Engineering space these days.
This is an interesting question indeed, is it not? What to use, what to use? Both DuckDB and Polars seem to be flying high at the moment, the new cool kids on the block. Everyone talking about them, and 1% of the people actually using them. Typical.
This is an interesting one indeed, it’s one that teases and puzzles the brain to no end. Has the Data Warehouse finally died, has that unruly upstart the Lake House finally taken its place atop the seething mass of data we call home?
I’ve been hacking around with tools and programming since Perl was a thing. I’ve worked the gambit of Data Platforms from large organizations to tiny startups, and all those in between. I’ve worked on Data Platforms that dropped ungodly amounts of money on SAP products, and places where we would build our own massive data processing platforms on Kubernetes.
It’s been a while since I kicked the tires on a new tool, you know I love to pick and poke at things. Also, in my never-ending attempt to keep on top of the changing landscape and keep the wolves (newer, smarter, younger engineers) from snatching my throne, it’s imperative I at least attempt to keep up with the times.