Recently, for some unknown reason, I was pursuing the new Stackoverflow … called Reddit, for Data Engineering … and I ran across an interesting question … more or less it was related to “what makes a good Software Engineer … in a Data Engineering context.”
I feel like I’m sitting on the edge of the ocean, not a nice wonderful sandy beach where the gentle waves are lapping at my feet and the breeze blowing through my non-existent hair, but rather a boiling and roiling ocean pounding the shoreline and destroying my beautiful castle of sand.
I’ve had something rattling around in the old noggin for a while; it’s just another strange idea that I can’t quite shake out. We all keep hearing about Arrow this and Arrow that … seems every new tool built today for Data Engineering seems to be at least partly based on Arrow’s in-memory format.
Kubernetes. K8s. Man, I remember those days when it was hot hot hot, and you were an idiot if you weren’t deploying something onto Kubernetes. If you didn’t know how to use kubectl then you were a prehistoric human, almost as bad as those SSIS users.
When I was young and full of myself, writing Perl and PHP, while your ma was still reading you a bedtime story and giving you a stuffy to fall asleep with, I had to program uphill, both ways, in the rain and snow. Not like you milk toast Data Engineers clickty clicking around Databricks and Snowflake UIs.
Ever since the days of yore, SQL has been the bread and butter of everyone from the old school DBAs to the new school Data Engineers. Of course, Python is probably the bread bag if SQL is the sandwich, or is it the other way around?
You probably think this is another internet clickbait title uh? Just trying to get you to clickty clickty and sell you some Google Ads. Two problems. I don’t have Google Ads, and I know a small percentage of people will actually listen to this advice. Whatever. There is a reason some developers struggle to move past the Senior role.