Hacker News Reader

I agree with your first point, Scala allows writing incredibly performant concurrent code. However, if -like me- you can't bear the thought of working with Scala anymore, Rust is an excellent competitor.

Note, I spent around 5 years working with Scala full-time and grew a profound aversion for the language and its ecosystem. It definitively affects my judgment. I found in Rust all the "good parts" of Scala minus the things I resented.

what did you resent?

In no particular order:

- the JVM

- the tooling for writing with VSCode

- SBT

- NullPointerExceptions bubbling up from Java libs

- the split between the functional programming community and the "Scala is just Java without ;" one

- everything related to implicits (creation, resolution, etc) and everyone using them

- the ability to write the same 0-100 loop with 4 different keywords and 3 different styles

- the inability to enforce a coding style properly and consistently with a team of more than 1 engineer

- too much people writing DSLs for no particular reason

- ++, +:, :+, :++, ++=, _+_, ::, :::, _, `, <%, %, %%, %%%, <-, =>, :>, >:, (╯°□°)╯︵ ┻━┻, ., .., ...

- compiler errors

- compilation time

Seriously.

I modify a single line, in a single file, and the scala system takes 2-3 minutes to recompile at work. Apparently it spends over a minute inferring types!

Hello! The types are EXACTLY THE SAME AS ALWAYS. Please cache the whole typing phase as a build artifact based on file/directory hashes. Maybe even make this cache source-control-safe.

I am curious if you are using SBT incremental builds? (putting aside how hard it can be to get and keep SBT working for a project ... I was the guy that had to do that so I know that pain quite well). Because in my experience with really large scala projects that should not be the case, unless you modify a file at the very top of the dependency tree.

We used what I would call the package object predef pattern, which basically involved, rolling your own predef to have certain functions / extension classess / objects always in scope through package objects added to each sub-project. And the build would cross-compile both jvm scala and scalajs. It would not take more than a second or two unless one of those predef files were touched (which would fire off effectively a full rebuild).

Well, neither I, nor anybody at the startup I work at apparently (it would seem) has been able to figure it out.

Out of curiosity, how many hours do you think it would take to get the compile time down? I wonder if we could find somebody to help us with that.

At first I would try sbt-tmpfs, and then the largest factor in compile time imho are dependencies. Make sure that you split your project in submodules of semantically valid units. That reduce the amount of code that has to be analyzed. And then make sure that you don't have many unused imports.

In every project I've been are some former Eclipse users that are accustomed to collapsing all imports and adding new ones automatically. They never look at their ever growing list of unused files that are searched, loaded, and parsed.

It kind of depends on what you are looking for. It sounds like it's improving compilation times while you develop so you can get errors messages from the compiler and so on faster which SBT incremental compilation should help with.

Are you guys currently using SBT? If so, is incremental compilation not working for some reason (or are you not aware of it?).

In terms of time, it would really depend on the size of the project and what is currently in place. It could be a couple of hours to a few days.

BTW my email is in my hn profile you want to discuss more privately.

Yeah we are using SBT. I'm off this week, but next week I can try to see if this has legs. I checked your profile but didn't see an email.

Upgrade your sbt version and make sure that whatever file you update is not (transitively) imported/used by every other file. If that doesn't help, check for excessive macro/typelevel usage.

Incremental compile times should be seconds not minutes, something is probably wrong with your setup.

There have been some great improvements in Scala tooling in recent years. Li Haoyi talks about all these new tools in the Hands-On Scala programming book. Some specific responses:

* tooling for writing VSCode: the metals project is great: https://github.com/scalameta/metals

* NullPointerExceptions: those should be handled with Option/ Some / None

* the inability to enforce a coding style properly - scalafmt is great for this and provides a "Go-like" automatic formatting experience https://scalameta.org/scalafmt/

* SBT - there are now alternatives like Mill: https://github.com/lihaoyi/mill

Glad you're happier with Rust, but know that Scala is a lot better now if you ever come back ;)

Yeah, the symbolic method names can be quite annoying and (often) make code unreadable. Especially ScalaZ went really overboard there.

I'm happy to say that the convention is becoming more strict in this regard and symbolic method names are (mostly) discouraged [0].

[0] https://docs.scala-lang.org/style/naming-conventions.html#sy...

I wrote Scala for three years or so professionally and agree with most of your points. A couple of comments though that haven’t been addressed by the sibling commenter who also addressed some of your points:

- JVM is JVM I guess. You either love it or you don’t. I personally really enjoyed the access to the ecosystem so that was a major boon for me.

- Gradle is just so much better than SBT in almost every way aside from simplicity. I found building Scala projects with Gradle to be quite straightforward and nearly every project that supports SBT also supports Gradle.

- I agree about VSCode tooling but I do want to establish that there are actually really nice tools for writing code in Scala - specifically Jetbrains is quite great. I am not saying your point is any less worthwhile, only that I don’t want someone to read your comment and take away that there are no good code assistance tools for writing Scala.

- Agree with the Java NPE’s. This is annoying

- Entirely agree with implicits. The Scala team either needs to better educate people on how to use them or just tell people not to use them unless it’s a very specific circumstance. I cannot tell you how many times I’ve dived into someone else’s code and spent literally hours trying to coax the compiler to do something that was bizarrely prevented by a poorly thought out usage of implicits.

- it is definitely possible to write “Scava”, and I wouldn’t really recommend a team adopting it without someone having at least a decent background in functional programming. Otherwise you might as well write Java if you are going to write imperatively.

- Yes, I find that the usage of symbols in Scala is a bit excessive, but once you learn what they do it does feel more terse and concise. It doesn’t stray into the illegibility of Perl IMO.

Overall it’s a nice language, fun to write in, but a bit frustrating at times. I wouldn’t use it personally at home but it does fill a nice niche professionally and it’s absolutely great for writing Spark code.

We have recently switched from SBT to Gradle and I'm finding Gradle a lot more painful than SBT.

For starters, SBT had a nice interactive shell, while Gradle takes quarter minute just to list the available tasks, after which it's another quarter minute to get anything else done...

Gradle is undoubtedly slower. However it’s at least 10x more capable of doing various things around building your app like testing, building images, publishing, etc. Like if I needed to build, fat jar, test and publish a multiproject repository I would almost certainly want to use Gradle over SBT.

But Gradle is probably a bit overkill for a single entry point small app that does one thing and has minimal testing needs (though it can certainly do that!). You wouldn’t bring in a dump truck whenever a pickup would work just fine.

> Gradle is just so much better than SBT in almost every way aside from simplicity.

I picking this up, because it was also mentioned in a sibling comment. Let's continue this discussion.

It doesn't matter if we have multiple alternative to SBT, some more usable than other. Rust has one package manager (+build tool +distribution tool): Cargo. That's it. Whatever the project you just cloned and whatever the platform you're using, `cargo build` is going to compile it. Now, if every Scala project starts using a different package manager, that's going to be a problem. The best solution would have been to make SBT usable without a prescription from your doctor. It didn't happen. That's okay, programming languages are just tools anyway, and they get replaced too.

The canonicalization does happen with Gradle like with Rust, only at a higher (JVM) level. If you work somewhere that has Java, Groovy, Scala and Clojure projects being able to canonically clone any of them and build them with ./gradlew clean build is amazing. That forces your org to completely adopt Gradle but that’s not terribly difficult for a technically minded leader to enforce. Of course, that only applies in certain circumstances, and as you imply using the right tool for the job is the most important thing.

Scala is incredible for web apps.

I can write my React/JS front end [1] and highly concurrent, real-time scalable backend [2] using the same language, build tools etc.

[1] http://slinky.dev

[2] http://zio.dev

>ETL / data transformations. Python is a big player here - but for stable and performant data pipelines, I strongly believe Scala is the better choice. For explorative things, python has the edge though.

I disagree. And I've done a lot of Scala ETL work. The issue is that in Scala ETL work your data structures are either untyped (Spark Dataframe), case classes or typed tuples. If they're untyped then there's little to gain from Scala vs. Python in terms of type safety and a lot of overhead. If they're case classes then you're moving around a lot of unnecessary data fields and you've got the overhead of making fifty intermediate case classes yourself. If it's typed tuples then it's on you to remember which field is what as nothing is named.

Some sort of intelligent compile time case class subset generation would have helped the situation immensely. I think frameless (https://github.com/typelevel/frameless) gives you that power but it's a lot of shapeless black magic overhead and I've never seen it used in production (and probably various maximum field limitations and much slower compile times).

edit: Case classes also have (or had) a steeply increasing compile time memory requirement as you add more fields which is fun when your data is 200+ fields long. Almost as fun as having to make a 200 field case class to read in a single file of which you only need 50 fields but want to be type safe.

edit2: Scala is popular in ETL because the Hadoop ecosystem is JVM and Java used to be atrocious. So Spark (which wanted Hadoop compatibility) picked Scala because it was better than the alternatives. However nowadays Databricks is investing a lot more into their Python support from what I can tell and there's Python native competitors out there (Dask, Ray, etc.).

> they're case classes then you're moving around a lot of unnecessary data fields and you've got the overhead of making fifty intermediate case classes yourself.

Haskell developer here... What about case classes and lenses? Do they solve this?

As I understand it lenses don't change the underlying data structure. For ETL you need a way to basically say "the code only uses fields X, Y and Z so we will only load X, Y and Z during runtime." Automatically based on usage without having to keep updating your lens definition. Modern on disk file formats are columnar so they can very efficiently read subsets of the data. If your data has 200 columns than reading the 199 unnecessary ones can be very slow.

They could help with the intermediate data structure but some of them aren't subsets or trivial derivatives. So you really need an inline way to create single use case classes. I think frameless in Scala can do some of this for standard transformations but that requires the black magic of shapeless.

Spark in Python (and the untyped Dataframe API in Scala) compiles everything internally before running it to achieve the above. So it's trivial to have unit tests on empty data structures which "type check" your Spark code.

Another possibility is to use "typesafe heterogenuous containers" as described in Joshua Bloch, "Effective Java", 2nd edition and probably 3rd edition. The idea is to use maps with specially constructed keys, that give you type safe access to their accompanied values. That is better than completely untyped, you have a place to document stuff at the declaration of those keys and you can easily create runtime checks, that check for the presence of a set of well defined keys in the map. The fact that they are described in a Java book does not mean that they aren't a nice pattern in Scala and several other statically typed language with generics.

"For explorative things, python has the edge" -- I'm a lazy programmer, how can I get contextual auto-complete like I can with Scala (and other type safe languages)?

From my experience, auto-complete for Python shows everything (like JavaScript's), so I end up googling.

From my experience, for exploration of data, you usually use the functionality you already know and you visualize things as much as possible. So there is not a huge advantage for Scala in terms of autocompletion and such, but there is quite a bit of advantage for python for batteries-included, visualisation and speed of execution.

But yeah, I get your point, I find it annoying too.

For exploratory work you're in a notebook so your auto-complete usually has access to the exact object instances in question. There's also some good machine learning based autocomplete systems out there nowadays.

Well said. In some of my older comments I've shat on Scala (compared to Haskell) but after being forced to write Python for a day job, I've realized how good I had it with Scala. Luckily I'll be writing some again soon.