I know data engineers who know just Python and SQL who make $500k at Netflix | Hacker News Reader

I know data engineers who know just Python and SQL who make $500k at Netflix

twitter.com

18 pointsdonutshop31 comments

I guess my response is "OK" ? Maybe this is surprising for some people, but... why would it be surprising? Data engineers are usually working in a few ways.

1. With in memory datasets using Python Notebooks

2. With big data storage systems like Hive

3. With something like Spark or maybe even Flink

All of these tools were built for people who, for the most part, only need to know Python and SQL. And they do a bunch of interesting stuff with those tools. Also, I guess maybe it's worth noting that under the hood the implementations are almost always C/C++.

I guess maybe someone could mistakenly think that this is a good thing? Or a bad thing? Is that the point? Sorry I don't have Twitter so I don't know if there are further replies clarifying this. I'm sort of at a loss for the point because this is not surprising at all but maybe this is targeting people really early on in their careers who aren't aware - that's sort of the problem with putting a Twitter post here, it feels like HN is not the right audience, but their followers may be really green and exactly the people who would not be aware.

I don’t know if you clicked through to the link, but there’s some additional context in the original tweet:

> You don’t need to know the high performance languages to make a killing as a data engineer!

As such, the tweet seems fairly self-explanatory.

I did, yes, my post was in response to the full tweet. I think I'm just way outside of the target audience because I didn't know about this "you have to know high performance languages as a data engineer" conversation.

It comes from the Hadoop era of Data Engineering.

Which is that you were expected to know Scala or Java in order to build pipelines.

That all changed when Spark added Python support.

Could be data scientists, in which case I would probably describe their skills as:

"Know python and SQL and have post-graduate level skills in data analysis and mathematics"

Actually there's no reason why you couldn't get the skills before graduate school

The actual work is not the issue. It's getting a response from an application and then passing the interview that's the challenge.

I've applied to Netflix many times for many different positions and have never once gotten a response. I assume they are inundated with thousands of applications for every position.

If that’s the case, why do they pay so much?

I'd guess that's part of why they get thousands of applications per role?

TLDR; probably the same as how many people try out for professional football and the few that get in for high pay. They want the best.

There’s a correlation to quantity of applicants and pay. You can lower pay in high demand jobs but you’re lowering your funnel for the best applications buried in the bunch. This doesn’t matter for a cashier job since you aren’t looking for world class cashiers to disrupt your industry.

I honestly think it's because they don't offer equity. If you compare total comp against other large tech businesses, it's roughly the same. That still makes Netflix super attractive to work for, since equity often feels like a shiny fishing lure that may never materialize.

I agree. I've applied to approximately 50 data scientist/engineer positions over the last few years. I have not received a single interview despite having a PhD in a quantitative field and several years of industry and research experience, etc. This is odd since I typically get at least one interview for around 15% of the jobs I apply to.

I assume many employers are flooded with candidates too, but that does not explain the salaries. If the supply of candidates exceeds the demand, you'd expect the price (the salaries) to be lower. I don't know what to make of it, but applying to jobs in general just seems to be getting harder and harder as the years pass.

Full tweet text:

>I know data engineers who know just Python and SQL who make $500k at Netflix. You don’t need to know the high performance languages to make a killing as a data engineer!

This post's title is a little bit rage-bait without the last part.

This may be the first time I’ve been misled by a headline linking to a tweet. Why did this even get posted

Yeah it's just a stupid take. I swear half these twitter posts are just nerds who read one reddit comment a la "should I learn rust to get a good data science job" and they immediately assume that represents sentiment of the majority, and they must immediately shout into the void their profound insights. "I know guys, actually just 2, who earn a lot of money and they don't know rust at all!" Is that useful to the majority at all? A couple of extremely isolated cases with no additional context? Twitter being twitter I guess.

It's data. I won't say the tooling is irrelevant, but surely the important thing is understanding the domain and the finer points of statistical reasoning that regular people so often get wrong. I'm sure there are cryptographers who can't code their way out of a paper bag but are one of 50 people on the planet who actually understand how elliptic curves work. My wife is pretty invaluable at her job even though she's asking me dumb newb tech questions all the time because of her background in electrical engineering. Someone needs to check the 25 year-old hotshots who come in knowing all the hottest JS frameworks but don't get why it makes no sense to just take the user options for tasking an electro-optical satellite and port them over exactly to a SAR platform. You can type check to perfection, but if your type doesn't accurately model the thing in the world you're trying to represent, no compiler can tell you that. You have to actually know something about the world outside of software.

Actually the data is mostly irrelevant. The tooling and engineering are what's important.

I've worked across Finance, Retail, Insurance, Telco etc and more often than not the data requires deep domain knowledge which would take months of full-time education to understand. Nobody bothers with it.

Analysts and Domain Experts will figure out the calculations and pipelines whilst the Data Engineers are responsible for ensuring everything works.

I work with engineers who barely know python and matlab who make as much or more than I do; they are experts in other things. There are things beyond programming languages you can become a valuable expert in.

What are those things in your particular example?

I met a guy working for SpaceX today. Not much on the software side of things but definitely an expert on SDR. Makes over £365k a year.

Imaging and image quality

There's lots of subject matter experts that aren't expert coders though, this is just my corner of the world.

This isn't interesting or new.

When Python and SQL support was added to Spark many years ago, Data Engineers quickly embraced it.

It's standard amongst all Data Science people that you know both. But the people who actually do well in the industry know a lot more such as what the underlying infrastructure e.g. JVM is doing. As well as some other languages like Scala or R for more specific tasks.

Data Engineers in particular are expected to take full ownership over performance and reliability as well. So more than a little bit of cross-over with DevOps and SRE.

Life pro tip, when deciding between features to implement, if one of those features is a dashboard. Do the dashboard. Executives love summary information.

Obvioulsy, they also know about data manipulation and analysis.

What coding skills does your MD have? How much do they make?

Specialized and in demand skills bring high salaries...

Ironically posted by a guy who walked away from big tech and bigger salaries after he got sick of it.

There's not much you can't do with Python and SQL for data engineering.

It's culture + networking. Whether one knows <language> or not is not correlated w/ salary.

Data engineering isn't about a particular tech stack...at all.

You can do anything in Python, and do it fast, as long as you don’t need it to actually run fast.

That is a completely acceptable scenario for a great deal of data analysis.

If the problem fits into the pandas/numpy/scipy framework, it runs fast, because all the math code is in Fortran/C/C++ and the Python is just the top level control.

I didn't know that! That explains a lot about why those tools are so popular.

Reply on news.ycombinator.com