Scala at scale at Databricks

> who both understands those advanced concepts and is not a FP fanatic.

Oxymoron.

> Scala the language is quite fine, but it attracts this kind of programmers.

It does. I call them "code scientists": engineers want to learn a language to build things, scientists want to build things to learn the language. I've interviewed >100 Scala engineers and if the panel thought they were the kind of person that just wanted to understand FP instead of solve problems, they were rejected.

I am now retired, but Scala is still my goto choice for engineering software. It checks all the boxes on concurrency, bulk I/O, ecosystem. In the hands of a sane and pragmatic practitioner, it's hugely productive.

I sympathize with your thinking. There are two "readabilities".

1. How easy it is for an average programmer to understand the code.

2. How understandable the code actually is if the reader is fluent with that prior knowledge.

If one finds the latter is much better than code written in conventional techniques, but people are scared off by the former, it can get pretty frustrating, and "people are stupid" may be a conclusion drawn.

---

But my experience is that those techniques do not yield better code.

And "Educate yourself" made me barf.

If all you do is add numbers together, is it anti-intellectual to question why one needs to grok Principia Mathematica first rather than just doing some arithmetic?

In other words, do you solve problems that actually require category theory or are you just navel gazing?

For most tasks programming it's more an engineering discipline than an intellectual pursuit. In that context complexity must always be justified.

I know, and have been told, to have written hard to read code. In my case it's usually vectorized code in numpy or a C or cython extension for really hot code. But I always have a good reason, usually performance when it matters.

It's usually said that premature optimization is the root of all evil, but it's nothing compared against premature abstraction. Optimization at least is local in its nature while abstraction tends to expand all over a code base, and when the assumptions made for the abstraction no longer hold people is still forced to keep dancing for a music that no longer plays.

"Educate yourself" assumes that your interests are everyone's interests but this is a vast field and not everyone has interest nor time to learn about category theory. Most probably your coworkers are quite intelligent. If they are not interested in your idea it may be that it's not appropriate or maybe not correctly framed, or not mature.

This ain't anti-intellectualism. Scala fanatics (more like FP fanatics who use scala) write overly completed code just for purity's sake. Its ok, but it creates a maintenance nightmare and a huge hiring problem.

Everytime the “go learn category theory if you want to become a better programmer” trope arises on HN (less often than it used to, but still occasionally) I’m left scratching my head. I actually learned category theory as a math grad student, before changing fields and going in a more applied direction. I’ve spent decades writing software since then, and I can easily think of about 10 areas of math and CS that are way more useful day to day. Once you’ve mastered calculus, linear algebra, probability theory, statistics, numerical analysis, optimization, algorithms, complexity theory, combinatorics, asymptotic analysis—sure, go pick up category theory.

Thank you for this reply. It demonstrates exactly the mindset of those few FP die-hards that I have seen wrecking havoc in a couple of companies where I worked for.

I'm not saying FP is bad, it is actually really powerful when it is in the hands of those who understand when and how to use it. The problem with the attitude that you demonstrate in your reply is also not limited to FP: in the era of OOO, there were those that spread the evangelism of design patterns to all places regardless if it made any sense; before that in the 90's there was a group of programmers that liked to generate code until no colleague understood anymore what was happening. And when you tried to tell them that abstractions don't come for free, but with a cost because they make code harder to understand, the answer for the last 25 years has always be the same, although rarely said straight: that it was only hard for -you- to understand, and not for the enlightened master himself.

I have seen more projects fail because of too much unneeded abstraction than by all other causes together. I even have seen companies go almost bankrupt because of projects engineered by lone wolfs where nobody understood the abstractions anymore except for the designer himself, and at some point he himself not really anymore either.

And the problem is that this mindset continues to be cultivated by CS books and conferences. Few people want to read a book that tells them that the secret of being productive as a team depends on mostly the culture in the team and the simplicity of the code, and not on the latest hyped framework, language or paradigm.

The thing is I DO understand category theory and FP after going off the deep end in Scala land. It did not make me a better programmer or make writing code any easier or faster. In fact it makes me spend 50% of my brain power on juggling those concepts and how they are encoded into a language that does not and never will support them as well as say Haskell.

I learnt the basics of FP a long time ago and understood the data-centric view of computation, the benefits of immutability etc - those things actually do help me write better code. The advanced FP, not so much. The nightmarish encoding of advanced FP shoe-horned into a language not built to support it that compiles into Java definitely not built to support it actively hurts. There are so many dirty macro shenanigans and hacks in the guts of the FP-purist libraries to force the language to implement things it just isn't particularly well suited to implement, which makes reading or debugging library code a nightmare. Like if you want Haskell, just use Haskell FFS and let us code reasonable Scala in peace.

There are so many parts of math that are much more useful in various fields - basic optimization in solving sudoku puzzles[1], linear programming in Z3, gradient descent in ML, calculus & vectors in graphics programming etc.

[1] https://norvig.com/sudoku.html

This is development nowadays, hire people fresh out if am eigh week javascript boot camp and everyone has to write code they can understand.

Instead of bringing the juniors up, we drag the more experienced guys down to their level.

The result is really shit basic code everywhere. or you end up using a language like go where there's basically only for, if and arrays and writing code in it is miserable and tedious.

There are others options besides Go and category theory. I'd say that for now, languages that are like ML are enough. They're coming in the mainstream with pattern matching, immutable records, etc. It'll take some time for people to learn this, learn to use it properly, teach it to others. And maybe some years later a monad will be a pattern as common and well-known as an iterator. But today isn't that day yet, and until now you have to collaborate with your peers. You also have to assume that you may be wrong about some of this stuff, and keep an open mind for FP alternatives. For example, I think algebraic effects aren't part of category theory, yet they're an exciting new feature that might have an impact in "industrial" languages one day.

The code base had multiple problems, none of which I would blame on category theory or Cats.

- Engineers had written higher abstractions seemingly just because they could. When I audited how internal libraries were used across our services, calling applications weren't making use of the advanced abstractions. I'm talking about things like using Cats to abstract across AWS S3 error handling. Cool, except that it wasn't needed because we never actually encountered the exotic compositions of failures anticipated by the libraries.

- The abstractions written for our own business logic were worse than abstracting over S3. They were premature. Our business logic had to change frequently because the end user experience was still evolving rapidly. Changes that violated previous assumptions and their corresponding abstractions took longer than they should have and/or led to very awkward code.

- At least at the time, tooling had more problems with the "advanced" code. The IntelliJ IDEA Scala plugin could not yet show how implicits were used. It couldn't find senders sending to an Actor the way it can easily find plain callers of a function. You would need to manually force a "clean" in certain modules before code changes would compile as expected. IDEs would also fail to flag code that couldn't compile, and incorrectly flag code that would compile, at a higher rate compared to plainer Scala.

I'm still glad that I have access to Cats, Akka, and other advanced parts of the Scala ecosystem. They're still used in a few places where their value is greater than their cost. Even in the plain code, I'm still very glad I have pattern matching, immutability-by-default, rich collections, map, flatMap, filter, fold, scan, find, etc. I have no plans to transition our company off Scala internally. If I were starting a greenfield project with myself as the sole developer, I'd probably be using Scala for that too. But I prefer to write a bunch of simple repetitive code first, then develop abstractions after it's clear what the commonalities are.

How did that conversation go?

“We like your solution but did you seriously just use Maven?”

What did they say exactly?

I also prefer to use Maven to compile Scala code - it's much faster, better integrated with IDEs, etc.

As opposed to Gradle or SBT?

If I'm being honest here, a quite significant part of day-to-day backend JVM software development positions is dealing with the build tool. While I wouldn't outright reject a candidate for not being familiar with the latest standard toolchain (Gradle), I would certainly consider it as one of many factors - Gradle is pretty complicated, yes, but also quite powerful in what it is capable of achieving. It's not as eyebrow-raising as, say, if you sent in a coding challenge compiled using Ant (which would, in fact, be quite a fun discussion to have with a candidate) but it would certainly be something considered.

That's silly but unfortunately this type of silliness is the norm in the industry.

> Scala has a unique way of encouraging genius programmers to write crazy code.

Scala is a popular platform to explore FP concepts without fully understanding their caveats or having enough respect for the rest of the team's ability and willingness to follow, I'll give you that. But that's only because it's a great platform to explore FP concepts in general!

You have to reign in the "geniuses", that's what engineering culture is for. I really don't like that label for people writing crazy code, btw. They shouldn't be on a pedestal just because they lack the empathy to write maintainable code.

The pythonic-Scala approach is certainly gaining steam in the ecosystem. The problem is, it's hard to see because it's boring and doesn't require many special tools – Scala has much of that built-in. So the popularity of this approach is not apparent if you just look at open source activity. But internally companies who work with Scala use it in very benign ways for the most part, in my experience at least.

In what way is Scala a better Python? The 2 seem nothing alike to me.

> Scala programmers generally frown on programmers that want to use it as a better Python and I think it should be the opposite.

It's hard to make generalizations like this that are meaningful. There are a number of different Scala ecosystems, as you probably know. Spark is really a world of its own. Lightbend's libraries form their own ecosystem, as do Typelevel's and ZIO's.

My own opinion is that using algebraic data types (sealed traits, final case classes) with a sprinkling of type classes (implicits) where needed is pretty easy to work with, and that's the style that Typelevel is using. It is mostly inline with Li's "Least Power" post, though using IO pervasively instead of Future.

Is Typelevel's approach a better Python? I don't know. I don't really know what that means, and I'm not convinced that Python is a great model to follow. I see you posting along similar lines in many Scala threads on HN, so you seem quite invested in the language. I think it would help make your point clearer if wrote up a bit more of your thoughts, perhaps as a blog post.

> Save yourself, don't use Scala, pick Rust.

That's an hilariously bad advice if you think hiring competent developers and onboarding juniors is hard in Scala.

Most things companies do on the JVM, especially at scale, are going to be a lot harder in Rust and its ecosystem.

I haven't used Rust but my understanding is that memory management is explicit, which doesn't sound great for general business applications. I've used Java, Python, JS, Kotlin, Go and Scala, and found Scala to be the best at representing domain models in a type safe and readable way.

When I say more freedom, I primarily mean freedom to focus on the problem domain and freedom to contribute in a major way more quickly. Upon reflection, I realize that my expression about optimal solution was about optimality in modeling and understanding the problem domain vs. wrestling organizational programming pattern you're trying to learn / hold in your mind while also trying to solve the domain (which you don't have to do as much with scala's compiler)

And in terms of morale, I view almost the opposite of what you're saying. Because types and the compiler enforce correctness in the program and compatibility within our systems I can have a relatively junior person thinking more about the domain and knowing that they can get something "real" into prod much more quickly with almost no risk to the org. They can move on to bigger blocks of work more quickly. Solving real, big problems and making impact gets engineers jazzed up. Regarding compilation, I find the "this won't compile" phase is almost non-existent in practice. People quickly adapt from "this thing doesn't compile [insert sad longing for python]" to "this thing won't compile - somewhere there's a misalignment with what I said I'm going to do (via types) and what I'm actually doing, [insert silent acknowledgement that they would have shipped an error in python and not known it]."

I'm not seeing where the gap is between A and B except that in option A it appears that there is no compiler involved, which per the above, means that I have to ask them to focus more on exercising mental discipline about coding conventions (boring, low value add work) and also I can't allow them to contribute as quickly because the risk they threw an exception somewhere they shouldn't have (e.g.) is higher.

Yea, there was a JSON parser in the Scala standard lib at one point and it was then deprecated & removed. Guess it's easy to make a "hello world" JSON lib, but maybe not the best addition to the standard lib, haha.

I think no, because, IMO, Kotlin doesn't have a good pattern matching like Scala.

I don't think so. Kotlin is an unmaintainable soup of features: it can match all the selling-point examples of Scala but only by adding ad-hoc special cases for each one. (For example, Kotlin has null safety and it lets you write code using errors-as-values style "either" types - but it has two completely separate syntaxes for these things, and so it's impossible to interoperate or reuse code between those two approaches). In practice Kotlin codebases still use magical incomprehensible reflection (Spring Boot) and magical compile-time manipulation (Kapt), so you have the worst of both Java and Scala worlds.

Yes, over-engineering is common in many languages, especially Java, where Spring and OOP patterns would be deployed everywhere. It is not a unique problem to Scala.

Scala moved the needle five years ago and made java better among other languages. Using scala helped me grow as an engineer. in my view, it has no place in an enterprise. I require a mature environment that favors consistency. It boils down to reducing complexity. I can barely handle the complexity growth due to the demands of the business, whether it be software engineering or people and these reasons do not justify scalas pluses. Ymmv.

I wouldn't say spark is terrible, but it is a sort of a "reverse bicycle". If you put a spark codebase down for a week or two and pick it back up, it feels like you're learning everything all over again.

> it seems moot to use it now

It's true that some features of Scala have made it to Java, but I am not sure the gap between the two languages has shrunk much. For example while Java has added first-class functions and closures, it is still clumsy to attempt any serious functional programming approach with Java; Scala's type system is still way ahead; Scala compile to the JVM, JavaScript, and now native via LLVM.

> having libs tied to what version of the compiler you're using

It's never been that. However libraries used to be tied to the major version of Scala (for example 2.11, 2.12, 2.13) as major versions were not binary-compatible with each other.

Scala 3 addresses this thanks to the TASTY format which creates a strong foundation for binary compatibility. [1] You can use Scala 3 libraries with Scala 2.13 and, more importantly, Scala 2.13 libraries with Scala 3.

In short, Scala is not standing still.

[1] https://docs.scala-lang.org/scala3/guides/tasty-overview.htm...

> In my previous company we used Scala + Spark and I hated it initially. I had no idea how syntax worked, no clue about functional programming or the ecosystem.

It's funny that if you tell Cat-purist that Spark is functional-style, they would laugh at your face :)

Scala + Spark is absolutely the sane face of Scala. Try using stuff like cats (basically Haskel ported to Scala) to experience the deep end of the pool.

I don't agree with him, but that's a biiit smug..

Regarding the reduction in lines of SQL. Doesn't this imply a tradeoff with where compute is utilized? By reducing SQL, wouldn't you be "pushing up" the compute from the database to the JVM while also increasing the amount of data that has to be moved around?

Edit: Wait, did you imply that the SQL you replaced with Scala ultimately gets transpiled into SQL (spark or other dataframe engine) and "pushed down" to the database? Now you're just maintaining Scala and not a rats nest of SQL strings concatenated together? If so, that's pretty awesome.

> after getting sunk in total complexity

Plus, the cost of teaching Scala to your new hires means they won't be productive for literally _months_

I agree that other big data tooling providers are eating into spark but I'm surprised people are choosing python over Scala for Spark.

Spark wasnt even released 10 years ago :) I think Twitter, Linkedin, and some others gave Scala adaptation big push. Twitter in particular, when they killed their fail whale by moving to Scala/JVM

Why not just use Jackson? Sure, it doesn't have a clever DSL. But then, that's a feature.

I worked at a company full of Scala zealots and elites. It was insanely unproductive and people spent a lot of time trying to design around all the floors in Akka, trying to get Akka working in modern cloud computing environments and the like.

I think at some stage specialists from some Akka pro shop had to be air dropped in to try fix it all, but it didn't help much.

It always made me laugh how well all the services written in Python and Go worked and how easy they were to extend, operate and deploy comparatively.

The standard library is where modules go to die, just like in Python. And unlike Python, there's a good build/dependency manager available for Scala (Maven). Keep that stuff out of the core language, let it be libraries released on their own schedules.

Well, you can always use tried-and-true Java library like Netty, Jackson, and plain-old JDBC API. You get to enjoy all the goodie like `val`, `case class`, `pattern matching`, etc... without all the headache of those pure functional libraries.

Those are the least confusing libraries in the Scala ecosystem lol. Slick is just like any ORM-light framework. Akka-HTTP abstracts away all the Akka-specific things and you just create routes that return Futures essentially. How is spray-json complex? You're just using macros against your case classes to create (de)serializers.

If anything, you should be saying things like Shapeless provide the extra complexity.

BTW, the described organisation does not hire "Scala Engineers", but Software Engineers. Scala is just a useful tool.

F# has some nice features that scala is lacking, such as a more terse syntax for ADTs or type providers.

But other than that, Scala is miles ahead. And if you want to do pure FP, then Scala is the only option.

Also about that video, almost all of the points mentioned are either not available in other languages like F# anyways, or they already fixed in the latest Scala version.

> Scala is the worst of the category of "functional looking languages built on the top of an imperative ecosystem".

Could you elaborate? Because that's not been my experience at all. I have found Scala to provide many tools to wrap around the idiosyncrasies of the underlying imperative that they are barely noticeable anymore.

We use TypeScript not Scala, but I've already found some other TS projects that use monorepos for the exact same reason I would do.

Still searching for pros/cons with current toolings etc, as most articles about monorepos are 3-4y old and therefore outdated in some points

Depends on what you're trying to do. For example I think that Lightbend is no longer planning to maintain Play. I would probably go with Spring as a replacement.

For Akka, do you really need it? You have to be building a pretty specific kind of system to need it over simpler constructs built into the language.