Scala Data Pipelines for Music Recommendations at Spotify
slideshare.net
slideshare.net
My team has plans of moving away from Scala and towards Python primarily because of the job market. There is a feeling that it will be significantly easier to find a good data engineer who uses Python than it will be to find the same who uses Scala.
Hiring experience with a specific language is of course easier with more popular languages. But unless you're hiring emergency consultants or something, it's more important to hire smart people who'll be able to learn whatever language comes up. And I suspect that while it'll be very easy to find someone who "knows Python", the talent level is... variable. Whereas people who know Scala, though harder to find, will be the kind of people who learn multiple languages, who enjoy learning new skills even when there's no obvious market for them - i.e. the kind of people who you want to hire.
We're currently looking to hire a second engineer and have also been rethinking the direction of the data engineering team. Changing the language to Python was a suggestion for the hiring factor and because it is the popular language for data engineering. Scala has been a fun language over the last year and a half and I wouldn't mind sticking with it because it has been fun. For that reason, I have spent time comparing the two languages for data engineering and haven't come to a consensus on which one is ideal for my team.
Are they able to match salaries with Google? :)
It depends on the level of the developer you can find and are willing to pay for. Not every dev shop is able to hire top talent, simply because that talent is limited (by definition) and not always necessary.
Most shops that are adopting Scala are doing so because Java has been so stagnant and they need to stay on the JVM. The argument of Python vs. Scala is less clear.
It's not always easy to understand (from Odersky himself https://gist.github.com/odersky/6b7c0eb4731058803dfd#file-fo...):
def toVector: Vector[B] = fold(Vector[B]())(_ :+ _)(_ ++ _)
To a Scala veteran, I'm sure that's easy to understand; to someone who's been learning the language (like me) it looks like gibberish.
I've also had to write code like this:
client.post(args).mapTo[Response].map(r => (r.success, r.serverException, r.unhandledException) match {
case (Some(response), None, None) => response
case (None, Some(serverEx), None) => throw serverEx
case (None, None, Some(unhandledEx)) => throw unhandledEx
})
simply because of the underlying api that other co-workers have built. When it takes you longer than 30 seconds to explain how a piece of code works to others, something is wrong.
All that being said, I do like the language; just not the compile times. It's also pretty close to ruby (with which I'm most familiar):
ruby:
numbers = [1, 2, 3, 4, 5]
numbers.select { |n| n >= 4 }
# [4, 5]
Scala:
val numbers = List(1, 2, 3, 4, 5)
numbers.filter(n => n >= 4)
// List(4, 5)
> To a Scala veteran, I'm sure that's easy to understand; to someone who's been learning the language (like me) it looks like gibberish.
Compare what the same code would look like in Ruby; something like (hope I get the syntax right):
def toVector = fold(new Vector()) {|x, y| x :+ y} {|x, y| x ++ y}
(You can think that's a good or bad function to have, but that's a library question, not a language question).
> simply because of the underlying api that other co-workers have built.
You can write bad APIs in any language; with a more sensible one that would look like:
client.post(args).map{
case Success(response) => response
case ServerError(ex) => throw ex
case UnhandledError(ex) => throw ex
}
I'd focus on hiring the best data scientists you can, and let them retool if they feel the need.
The use of Scala predates me, it's been going on about 5 years although there's been a huge shift in that direction over the last year. This time last year I'd say it was about 50/50 Scala and Java and now almost all new code is written in Scala. I've been told that the initial "why" of scala was that it made it easy to write DSLs with which to write pipeline data flows w/o leaving the JVM. We have our own pipeline infrastructure written in house (in scala) although we're starting to move toward more OTS options (also in scala).
The surge over the last year has more to do with personnel changes within the group with more people joining who were more pro-Scala (such as myself), mainly for some combination of the functional trappings and the reduction of boilerplate and other nonsense from Java.
In terms of the job market, while we'd love to see a candidate who is a Scala wiz, we don't expect them to have an ounce of scala background. Personally I don't care if they have any java background either, although not everyone in my group agrees. I lean very heavily on the side of hiring smart people instead of hiring for buzzwords, assuming that we're not talking about contract positions.
FWIW the data scientists in our group are mostly split between R and Python, with some also writing a fair amount of scala. The latter has more to do with some of the in house pipeline management tools than any data science-y reasons.
This should be number one!