Hacker News Reader

Comment by dominotw | Hacker News Reader

I've been in this space last 6 yrs or so and my scala usuage has gone down to zero. Not worth learning scala.

This is an anecdote - plenty of firms are using Scala in their data engineering stacks and it's a great tool for the job.

While maybe not strictly necessary per se, it's a great way to get a foot in the door, and provides a great way to foster advanced type systems and functional programming (I personally find it to be a really fun language to write in to boot).

> it's a great tool for the job.

What job can this do that can't be done via sql. dealing with unstructured data?

> plenty of firms are using Scala in their data engineering stacks

Isn't that just a result of everyone being into Spark a few years ago?

Regardless of the source I think the main point folks are missing is that a lot of DE jobs will require you to know Scala so it's a good tool to have if you want to be a DE somewhere.

SQL is also an amazing tool and you should definitely learn if but there are a lot of DE jobs out there with Scala in the "Requirements" section of the job listing. Parts of the industry might be moving away from it, but if you're looking to make a jump into DE I think you're hamstringing yourself by avoiding Scala.

Scala, when it's not used because it's just what someone learned the ropes with, is the Haskell of data science and machine learning: it's what people use when they want to inflate their credentials and/or egos.

What languages are worth learning?

SQL has taken over the space completely. 90% of data munging and transforms happen via SQL.

I would learn python. Its the number one language outside sql.

What do you use if you need to process hundreds of GBs of data?

sql on snowflake

PySpark or Dask

Assuming the data is relatively clean in the first place - there is a lot of data sources where SQL would be an absolute swine to work with.

sql, python, terraform, maybe some basic java. Airflow is pretty common. Whatever the company is migrating away from. As long as you`re good at one of those and can pick up the rest on the fly you should be fine to start out.

edit: Guess this was pretty much in the post.

Reply on news.ycombinator.com