Hacker News Reader

Comment by mahathu | Hacker News Reader

What does a data engineer do?

Suffer mostly.

Joke aside, data engineers do a wide variety of engineering work related to data. Often that's around data warehousing and business intelligence, sometimes it's work on big data streaming or storage systems, or building yet another integration for marketing.

The breadth is a challenge, which makes it fun and also hard.

No, you had it right the first time. It's fairly dull, monotonous drudgery. It's not fun , at least for any unconstrained definition of fun, for the majority of work that needs to be done. I speak as someone who does this for a living for over 10 years. I'm in such high demand as an independent contractor I think about starting a consultancy but one of my big concerns is how to recruit people with a high tolerance for blandness, ambiguity, as well as high conscientiousness. I think the subject matter is not difficult but the personal attributes required to persevere and succeed are rare. For me it's been the "muck is brass" career approach.

It's digital plumbing, almost literally: the pipes aren't very interesting, and you have to often wade through shit to put em in

Thankless work so that the analysts can turn on the faucet and get nice clean data for their fancy algorithms and models

It's plumbing, but you're also expected to know where the water came from, the exact specifications of the treatment plant that cleaned it, exactly what is in the water down to the parts per billion, and have ideas for what kind of faucet will make everyone trust the water the most.

Also bonus if you know how to run a business and give guidance on what type of house to buy before putting in pipes.

Disclaimer: I'm in the process of making the switch to software development, and I'm from a 3rd world country.

Is there any chance for someone with my profile to get a job in this? Remotely of course, as any remote job would pay times better than anything I could get in my home country. If yes, where does one learn about it more? I see Coursera has some course, but better asking from someone with your experience I guess. My approach to a job is that you do it in order to get money to fulfill whatever you enjoy in life, as long as I'm paid I'll be the best I can. Thanks

What are the skill sets required? Proficiency in SQL seems like an obvious one but else needs to be in the toolbox so to speak?

Kafka is a very useful one.

Core is SQL + Python (or Java/Scala for the actual Engineering part).

Most of the data warehouse work is data transformation and scheduling, so Apache Airflow and writing DAGs in python is a large part. For some of the larger streaming things you've got Kafka and others.

Also every cloud has it's own stack that is sometimes worth using.

Thanks, what's the name of the AWS stack? Is it worth using?

Depends on where you work at.

A lot of the time they do ETL/ELT mostly with airflow nowadays.

Some use little more than SQL and if they're lucky something like DBT to template and schedule the queries.

A lot of SQL now runs on data warehouses of which snowflake and Google bigquery stand out, but it's SQL nevertheless.

A few lucky ones get to use Spark which is more interesting as it's written in scala or python, but from my experience 99% of the time businesses are better off training business analysts to use SQL and writing the queries in snowflake.

Another interesting area is real time data, for existing e with Kafka. This can be challenging and fulfilling except a lot of the time businesses people will tell you it's cool but they don't want the data to keep changing so could you please aggregate it every day... So you end up with an overly complicated batch system anyways.

For me the sweet spot of practicality right now is Airflow/DBT/python/snowflake.

> A few lucky ones get to use Spark which is more interesting as it's written in scala or python

If you want a fun language, it's really the wrong place to be.

The work can be very fun, but you must take the fun from data analysis. The best software engineering can do for you is to not get on your way (and SQL excels on this).

I'd say the people working with Spike are the unlucky ones. Because they have to focus more on their tools, and not on the analysis.

Previously known or associated with terms such as ETL or ELT or data warehousing or ... In my experience tends to be involved heavily with databases, data movement, data modelling. You might think of it in terms of the work that needs to be done before data analysis is possible. Though as with many things in data, labels are imprecise and considerable overlap.

Reply on news.ycombinator.com