Explicit term inference with Scala 3
scala-lang.org
scala-lang.org
- Safe and high performant concurrent programming. Scala is miles ahead of most other languages here and pretty much on par with the best ones (such as Haskell)
- ETL / data transformations. Python is a big player here - but for stable and performant data pipelines, I strongly believe Scala is the better choice. For explorative things, python has the edge though.
- Actor systems on the JVM. I believe that most "scalable" systems are over-engineered. A well configured postgres / elasticsearch and a machine with your application on good hardware goes _very_ far. But sometimes it's not enough and then Scala offers a great solution with Akka on the JVM. Erlang/Elixir with their runtime are probably even better, but Scala offers the better typesystem and one can stay in the JVM world if they want.
- in-language custom DSLs. No seriously, Scala is the best language I know in terms of building typesafe, customizable DSLs within the language, making it almost look like another language. This is great if not so techy folks are to make changes to actual code in an easy way. The link in this post here shows some examples but it goes beyond.
- Writing glue-code. Surprisingly, I think Scala is better than python and many other languages here, because of the sheer power of composability of the language and ways to connect pieces with each other in a typesafe manner. The reason for this lies mostly in the concept of implicits, which both enables great power and reusability but also makes the learning curve much steeper.
This list might be a bit subjective and I don't know every language out there, but as the questions comes up almost everytime, I wanted to list the strengths of Scala in the areas where most other languages can't compete with it.
I can write my React/JS front end [1] and highly concurrent, real-time scalable backend [2] using the same language, build tools etc.
[2] http://zio.dev
From my experience, auto-complete for Python shows everything (like JavaScript's), so I end up googling.
But yeah, I get your point, I find it annoying too.
Note, I spent around 5 years working with Scala full-time and grew a profound aversion for the language and its ecosystem. It definitively affects my judgment. I found in Rust all the "good parts" of Scala minus the things I resented.
- the JVM
- the tooling for writing with VSCode
- SBT
- NullPointerExceptions bubbling up from Java libs
- the split between the functional programming community and the "Scala is just Java without ;" one
- everything related to implicits (creation, resolution, etc) and everyone using them
- the ability to write the same 0-100 loop with 4 different keywords and 3 different styles
- the inability to enforce a coding style properly and consistently with a team of more than 1 engineer
- too much people writing DSLs for no particular reason
- ++, +:, :+, :++, ++=, _+_, ::, :::, _, `, <%, %, %%, %%%, <-, =>, :>, >:, (╯°□°)╯︵ ┻━┻, ., .., ...
- compiler errors
I modify a single line, in a single file, and the scala system takes 2-3 minutes to recompile at work. Apparently it spends over a minute inferring types!
Hello! The types are EXACTLY THE SAME AS ALWAYS. Please cache the whole typing phase as a build artifact based on file/directory hashes. Maybe even make this cache source-control-safe.
We used what I would call the package object predef pattern, which basically involved, rolling your own predef to have certain functions / extension classess / objects always in scope through package objects added to each sub-project. And the build would cross-compile both jvm scala and scalajs. It would not take more than a second or two unless one of those predef files were touched (which would fire off effectively a full rebuild).
Out of curiosity, how many hours do you think it would take to get the compile time down? I wonder if we could find somebody to help us with that.
In every project I've been are some former Eclipse users that are accustomed to collapsing all imports and adding new ones automatically. They never look at their ever growing list of unused files that are searched, loaded, and parsed.
Are you guys currently using SBT? If so, is incremental compilation not working for some reason (or are you not aware of it?).
In terms of time, it would really depend on the size of the project and what is currently in place. It could be a couple of hours to a few days.
BTW my email is in my hn profile you want to discuss more privately.
Incremental compile times should be seconds not minutes, something is probably wrong with your setup.
* tooling for writing VSCode: the metals project is great: https://github.com/scalameta/metals
* NullPointerExceptions: those should be handled with Option/ Some / None
* the inability to enforce a coding style properly - scalafmt is great for this and provides a "Go-like" automatic formatting experience https://scalameta.org/scalafmt/
* SBT - there are now alternatives like Mill: https://github.com/lihaoyi/mill
Glad you're happier with Rust, but know that Scala is a lot better now if you ever come back ;)
I'm happy to say that the convention is becoming more strict in this regard and symbolic method names are (mostly) discouraged [0].
[0] https://docs.scala-lang.org/style/naming-conventions.html#sy...
- JVM is JVM I guess. You either love it or you don’t. I personally really enjoyed the access to the ecosystem so that was a major boon for me.
- Gradle is just so much better than SBT in almost every way aside from simplicity. I found building Scala projects with Gradle to be quite straightforward and nearly every project that supports SBT also supports Gradle.
- I agree about VSCode tooling but I do want to establish that there are actually really nice tools for writing code in Scala - specifically Jetbrains is quite great. I am not saying your point is any less worthwhile, only that I don’t want someone to read your comment and take away that there are no good code assistance tools for writing Scala.
- Agree with the Java NPE’s. This is annoying
- Entirely agree with implicits. The Scala team either needs to better educate people on how to use them or just tell people not to use them unless it’s a very specific circumstance. I cannot tell you how many times I’ve dived into someone else’s code and spent literally hours trying to coax the compiler to do something that was bizarrely prevented by a poorly thought out usage of implicits.
- it is definitely possible to write “Scava”, and I wouldn’t really recommend a team adopting it without someone having at least a decent background in functional programming. Otherwise you might as well write Java if you are going to write imperatively.
- Yes, I find that the usage of symbols in Scala is a bit excessive, but once you learn what they do it does feel more terse and concise. It doesn’t stray into the illegibility of Perl IMO.
Overall it’s a nice language, fun to write in, but a bit frustrating at times. I wouldn’t use it personally at home but it does fill a nice niche professionally and it’s absolutely great for writing Spark code.
For starters, SBT had a nice interactive shell, while Gradle takes quarter minute just to list the available tasks, after which it's another quarter minute to get anything else done...
But Gradle is probably a bit overkill for a single entry point small app that does one thing and has minimal testing needs (though it can certainly do that!). You wouldn’t bring in a dump truck whenever a pickup would work just fine.
I picking this up, because it was also mentioned in a sibling comment. Let's continue this discussion.
It doesn't matter if we have multiple alternative to SBT, some more usable than other. Rust has one package manager (+build tool +distribution tool): Cargo. That's it. Whatever the project you just cloned and whatever the platform you're using, `cargo build` is going to compile it. Now, if every Scala project starts using a different package manager, that's going to be a problem. The best solution would have been to make SBT usable without a prescription from your doctor. It didn't happen. That's okay, programming languages are just tools anyway, and they get replaced too.
I disagree. And I've done a lot of Scala ETL work. The issue is that in Scala ETL work your data structures are either untyped (Spark Dataframe), case classes or typed tuples. If they're untyped then there's little to gain from Scala vs. Python in terms of type safety and a lot of overhead. If they're case classes then you're moving around a lot of unnecessary data fields and you've got the overhead of making fifty intermediate case classes yourself. If it's typed tuples then it's on you to remember which field is what as nothing is named.
Some sort of intelligent compile time case class subset generation would have helped the situation immensely. I think frameless (https://github.com/typelevel/frameless) gives you that power but it's a lot of shapeless black magic overhead and I've never seen it used in production (and probably various maximum field limitations and much slower compile times).
edit: Case classes also have (or had) a steeply increasing compile time memory requirement as you add more fields which is fun when your data is 200+ fields long. Almost as fun as having to make a 200 field case class to read in a single file of which you only need 50 fields but want to be type safe.
edit2: Scala is popular in ETL because the Hadoop ecosystem is JVM and Java used to be atrocious. So Spark (which wanted Hadoop compatibility) picked Scala because it was better than the alternatives. However nowadays Databricks is investing a lot more into their Python support from what I can tell and there's Python native competitors out there (Dask, Ray, etc.).
Haskell developer here... What about case classes and lenses? Do they solve this?
They could help with the intermediate data structure but some of them aren't subsets or trivial derivatives. So you really need an inline way to create single use case classes. I think frameless in Scala can do some of this for standard transformations but that requires the black magic of shapeless.
Spark in Python (and the untyped Dataframe API in Scala) compiles everything internally before running it to achieve the above. So it's trivial to have unit tests on empty data structures which "type check" your Spark code.
Though, I am curious if there is anyone else who shares my mindset on Scala.
In a corporate environment, I found it to be an extremely expressive and powerful language but that power comes at a grave price, which I'll try to summarize:
- it's difficult to understand other people's code compared to other languages (e.g. Go) - it's so very implicit that you end up having to hold a lot of state in your head to understand what's going on. - the language is basically impossible to read effectively without IDE features.
I usually enjoy reading most code bases, but Scala is downright painful to read in plaintext. You end up having to be a compiler.
I think it depends a lot on the code. Scala 2 gives you a lot of guns to shoot yourself in the foot (see: implicit, as discussed here) - but you can also use it in its mostly-functional form, with immutable values, and then it can become very readable. Granted - I've always been using IntelliJ but I could read the scala code on Github too, just fine.
IMO modern language designers should be making IDE-first languages - after all, most serious programmers do use IDEs (even if they build that IDE within Emacs or Vim). The problems with "visual languages" are that without a textual representation you can't meaningfully diff/merge/blame, not that using the GUI is an inherently bad idea. With Scala you get the best of both worlds: it's textual enough that version control works properly, but you have standard-ish way of folding, hovering etc. that mean that you can "zoom in" on the details of unfamiliar code but also "zoom out" to get a clear overview, in a way that few other languages manage.
Because, once I got used to the (pure) functional programming style, I found it more difficult to mix this style with imperative code. It's almost always easier to for example use for example the state monad than writing it with mutation. This is probably quite subjective and it depends what you are used to.
For me, I make an exception for tests. E.g. when I mock my key-value database with an in-memory map, I choose a mutable map. But I know that some people prefer FP style even for tests.
If I can't understand the type of a route (or a db-query) because it's a 5-level nested type, how am I going to write a function that takes one as a parameter?
</rant>
I'm no stranger to diving into a new codebase and trawling my way through spaghetti to make sense of it ... but I absolutely do not want to do this with a Scala codebase ever again.
Oftentimes one stares at a mile-long chain of methods on a list and is left wondering "exactly what did the original author want to do here, and why are things breaking"? In the end, the only way to find out part of the answer is to exercise the relevant codepath via some test data (if you're lucky) and use IntelliJ's excellent debugger to help you move forward. Even this just reveals the "what", not the intent.
No amount of consistent linting or formatting can make this better. And no, ExpressivelyVerboselyNamedFunctionsInAidOfSelfDocumentingCode don't help either.
I wouldn't use the word "state", since that has connotations that implicits are somehow changing at runtime. It's certainly true that scope is much more important and non-obvious than explicit variable names.
Type aliases can also get in the way when thinking about implicits (e.g. we might want an implicit 'Foo[Bar]', but if those are aliases then the compiler might start chaining Maps of Lists of Strings of whatever to resolve it).
Implicit conversions definitely seem like they could cause headaches (e.g. performing 'String' operations on a 'List[Char]' and having it implicitly and silently converted). I've only been using Scala for the last year, and thankfully implicit conversions seem to be avoided these days!
Scala is a beast and the package manager + build tools were giving me headaches. I’m optimistic about the current trajectory with Scala 3 and simplifying the language.
Ignore them and use maven. It's much better documented, consistent, and backwards-compatible. I struggle to understand why SBT ever gained any popularity, and it should certainly have never been recommended to newcomers.
Maven is powerful and people forget how innovative it was. But it’s not easy.
The page I linked to is pretty clear about which is which and what they do.
> And do I manage my dependencies in ‘dependencies’ or ‘dependencyManagement’?
You list your dependencies in dependencies, you manage them in dependencyManagement. But it's clear in the documentation, and if you're looking at a tutorial from 10 years ago then it'll still be accurate.
There are confusing things in maven. But documentation and backward-compatibility go a long way, and I really think the fixed build lifecycle puts it head and shoulders above a lot of alternatives; the cost/benefit of each project having a slightly different set of build commands just doesn't stack up.
Once you start to dig deeper, you find out that setting multimodule projects is still easier in SBT. And if you decide to dig deep and really learn the build tool, it doesn't really matter which one you choose. But devs that chose SBT as their Scala build tool will probably keep using it.
The build REPL has lots of commands that aren't documented, and when you search for tutorials the commands have changed (e.g. runMain is apparently now run-main, and if you write runMain you get the wonderfully helpful error "Expected ';'"). There's nothing so simple as Maven's list of phases https://maven.apache.org/guides/introduction/introduction-to... ; instead each project has its own slightly different set of build commands.
> Once you start to dig deeper, you find out that setting multimodule projects is still easier in SBT. And if you decide to dig deep and really learn the build tool, it doesn't really matter which one you choose.
Disagree. Maven multimodule projects are very simple: you can have a module that contains a list of submodules to build and... that's it. Importantly, every submodule acts exactly like a normal top-level project, so if you only ever want to work on one module you don't have to understand anything about the bigger project. And moving between single-module and multi-module projects isn't a big conceptual leap.
SBT multimodule projects are not only their own unique thing, they make the above problem even more confusing, because each project has its own slightly different set of build commands, and each module has a slightly different subset of them that works.
And even if you really learn it in detail, SBT is still awful on multiple levels. https://www.lihaoyi.com/post/SowhatswrongwithSBT.html talks about the deeper problems.
Maven multimodule projects are strange. Yes, modules can work without knowledge of what's above them, but that's not usually what we want to do, because it means lots of duplication (e.g. dependencies) which can then lead to inconsistencies. And if the common settings are defined within the parent project, it stops being self-contained. SBT accepts this reality and that's why multimodule projects are a thing. And having multiple subprojects in SBT doesn't mean that suddenly some of them work and some don't. By default, every project has the same settings (save for the obvious ones, like where the sources are). Any extra settings and task definitions just tell SBT to do extra things when certain tasks are run. One quick example: if you enable the integration test configuration just for some subprojects, the ones that don't have them don't start to fail. They correctly report a "successful, 0 tests run, 0 test failed". By the way: creating a separate integration test config in Maven is painful, up to the point that people recommend to just add a new subproject that contains them. Yay, more nesting!
Regarding SBT current shortcomings: that article correctly points out lots of issues, like the lack of tooling. To this day, IntelliJ still has issues downloading the correct SBT sources. And when they work, there's no easy way to find the actual code that does the work. That should be improved, because for lots of plugins, a quick look at the source is just what's needed when working on the build. Namespacing is a problem too. The internal complexity of the core SBT concepts is complained about too, but that's not as solvable as the other two things.
All in all, Maven and SBT are different tools that focus on different issues, and choose different sides in the consistency vs flexibility tradeoff.
Ah, I got it backwards: old tutorials say to use "run-main", and if you do run-main now you get "Expected ';'".
> There are lots of available commands indeed, due to the nature of SBT, but a basic workflow always uses the same commands: compile, test, run. Any extra tasks used usually come from plugins, just like in Maven. And reading documentation is needed for configuring and running those, just like in Maven.
But in Maven you don't need to read any documentation if you're just building the project, because every project has the exact same lifecycle. If you want to add or remove plugins to make the build do something different then you need to read the documentation of those plugins, sure, but you don't have to know anything about the plugins if you're just working on the code. E.g. if an SBT project is packaged for docker then you'll have to run some docker-plugin-specific command to do it (and you've got no way to discover which submodule you're supposed to run it in), whereas if a maven project is packaged for docker then you just run "mvn deploy" as usual.
> And if the common settings are defined within the parent project, it stops being self-contained. SBT accepts this reality and that's why multimodule projects are a thing.
Parent projects work consistently whether or not you're using a multimodule project, which is another problem with SBT - it's really confusing to share parts of a build definition between more than one project, to the point that most people copy/paste instead. With maven you can do a very natural, gradual progression from single module -> multi-module project -> multi-module project where the parent is its own module -> independent projects using a shared parent (e.g. an organisation-level parent). It's another case of a hierarchy working much better than a grid.
> By the way: creating a separate integration test config in Maven is painful, up to the point that people recommend to just add a new subproject that contains them. Yay, more nesting!
Think of the people who come to join your project! I've seen SBT modules with 5 different scala source directories and no obvious relationship between what depends on what, and good luck getting any IDE to understand whether test depends on integration-test or vice versa (most will give up and just build everything together, which is fine until you add something that works in the IDE and then errors when you build it with SBT).
A separate submodule is a much better approach - tools and people are much better at handling "module A depends on module B" than "this source folder depends on that source folder".
Regarding "discovering which submodule you're supposed to run commands in": this is a non-issue. Commands are either run under the obvious subproject (e.g. "api/run", "business-logic/test"), or are run as top-level commands. This is true 99% of the time. I have a fairly complex project open right now, and I just run "docker:publishLocal" to create two separate Docker images, one for each subproject that includes the appropriate plugin. I think this is better than overloading a single command. SBT projects usually have a small readme describing what is the command for any specific task you're supposed to do. In Maven, you know the commands, but you need to read to ensure what they do (does "deploy" push to an artifact repository, does it create a container image, does it create a deb package, or does it create a standalone jar?).
The use case where a multimodule project becomes multiple projects is more gradual in Maven, I fully agree on this. The problem, as I said, lies in that child Maven projects are not self-contained. They usually depend on information declared in the parent builds. This breaks the assumption that people can work on a project without worrying about the parent project.
Finally, source dependencies are explicit in SBT save for the base case "test depends on main". Everything else must be specified, so it's either there in the build file, or added as part of a build plugin. It can also be checked from the REPL. And building from the IDE is a dumb mistake if the IDE doesn't use the correct build tool underneath: the build is there for a reason and the IDE shouldn't bypass it.
"package" creates some kind of package, "deploy" pushes it to some kind of artifact repository. The details of what kind of package and what kind of repository will vary from project to project, but you don't need to know them to start work on the project. In theory there could be projects out there that do surprising things (after all, nothing actually stops you from configuring your build to push a container image to a repository when someone runs "mvn test"), but the overwhelming majority of projects find a way to fit themselves into the maven lifecycle, and it really reduces the "where do I start" effect when you check out a new project: I don't have to read the build file or hunt for documentation, I can run "mvn install" and see what it does, and be confident that that's "how you build this project".
> The problem, as I said, lies in that child Maven projects are not self-contained. They usually depend on information declared in the parent builds. This breaks the assumption that people can work on a project without worrying about the parent project.
Well, SBT projects also come with a bunch of default tasks that aren't defined explicitly in their project definition; some of them are defined by plugins and some of them are defined... somewhere (I genuinely don't know where the tasks that are available by default come from, or how to see a list of all of them except by using the interactive commands). So an SBT project is not self-contained in that sense either. In maven anything that's not explicitly defined in the project is coming from its parent pom, which is either the default parent pom or an explicitly specified one, and the default parent pom is a real pom that you can look at that follows the normal rules.
So you do have to understand the concept of a parent pom, but it's not an extra thing that you have to understand, because the default parent and any parent pom you're using in a multi-module project work exactly the same way.
> And building from the IDE is a dumb mistake if the IDE doesn't use the correct build tool underneath: the build is there for a reason and the IDE shouldn't bypass it.
Yes and no: the IDE has a bunch of knowledge of its own that the build tool doesn't, such as which files the user has changed, or which specific test the user has asked to run, or which import the user wants to add a dependency for. So I don't think the idea of the IDE dumbly invoking the build tool and letting it do its thing works (particularly in a language like Scala where you have to rely heavily on incremental compilation if you don't want enormous build times); instead there needs to be a deeper integration where the build tool and the IDE share a structured model of the project, and the IDE can perform a build via that model (effectively embedding part of the build tool as a library, if you like). Likewise I don't want to invoke a separate build REPL to understand the project definition, I want to be able to explore it within the IDE.
I believe the tagline/motto of every build tool should be “Don’t worry; You’ll get used to it.”
And they support Scala 2.13
I think Scala was actually a better choice since my company mainly does AI/ML (python) and Scala has some clout in data science. But I’m only migrating the application layer so not a deal breaker.
I was leaning more towards Scala going into the “exploration phase”.
——————————
Side note. Using typescript on front end, f# on back, vscode, and GitHub for repo + CI/CD. Feels nice using a single vendor (Microsoft) for dev, even though each tool is stand-alone.
This is the type of stuff that made me go away from Scala.
Why implement a feature that optimises for code writing? How does my editor let me know that function receives that parameter without me going "???" and having to go into its definition?
Implicit conversions fall into the same category, of making code pretty to look at, quick to write, and a nightmare to understand when you're new to a codebase.
Shame, really. The good parts of Scala are wonderful.
IntelliJ has been able to do that for a while now.
> Implicit conversions fall into the same category, of making code pretty to look at, quick to write, and a nightmare to understand when you're new to a codebase.
Don't do it, and don't allow it in your codebase.
It has been 4 years since I did Scala everyday, back then it was hit or miss. Good to know things have improved.
> Don't do it, and don't allow it in your codebase.
I get your point, but that doesn't solve the issue if the community is down for using these features.
And I don't think I've encountered implicit conversions that often in the ecosystem.
Every real-world Java codebase ends up using massive amount of incomprehensible magic for e.g. DI, transaction management, web request mapping, serialisation. Every real-world Python/Ruby codebase ends up using magic proxies, metaclasses, method_missing or other such stuff. Scala is the only language where I've been able to find real, enterprise-scale codebases written in 100% plain old code without any AOP-style magic, because the implicits let the language be expressive enough that you can avoid those things without your business logic getting drowned in secondary concerns.