Strategic Scala Style: Principle of Least Power
lihaoyi.com
lihaoyi.com
Haoyi says "if you are actually modeling something which changes over time, using mutable [variables] is fine". I think this is absolutely wrong. You should even use immutable variables to model things which change over time (after all, almost everything does). Only use mutable variables when performance means you can't get away without them.
Other than that the guide is pretty much how I would program (as a Haskell developer, rather than a Scala one).
The fact that Scala is proper multi-paradigm, and has a first-class functional sublanguage as well as a first-class imperative sublanguage is part of its great appeal.
I have mentored quite a few people over the last couple of years in Scala. In my experience, going from imperative Java, C, Python straight to Scalaz/Typelevel/higher-kinded monad style pure functional programming never works, and drives Scala-beginners away.
What works extremely well is to start with Scala as a nice Java without semicolons, and then, as they become more proficient in the OO/imperative Scala fragment, gradually to introduce more advanced features, starting with case classes.
The reputation Scala has for being too difficult comes from certain parts of the Scala elite trying to convince everybody, including beginners, that higher-kinded purely functional Scala is the only way to use the language. Nothing wrong with this style of programming per se and it's great that people are pushing the envelope, but the conceptual gap is too big for a language learner hailing from the imperative world (which is most of them).
I would agree with you if you could write something like
in a pure immutable style and show me how it's better. You will start off with lots of `.copy`s, maybe transition into lenses and other optics, and soon you'll be on the bleeding edge of research and still not have a working game that's maybe high-school difficulty if written in a mutable style
Research is cool and immutability is cool but if I can get what I want only needing some highschool kid instead of a PL-PhD that's cool too
I'm genuinely sorry if my comment touched a raw nerve. As I said, I think it's an excellent article.
One thing I got wrong in my original comment: use mutability where performance means you can't get away without it, but also where code complexity means you can't get away without it. (My personal view is that there are many cases of the former but few of the latter, but that's an argument for another day.)
"code complexity means you can't get away with it" seems very common. Five nested `.copy` calls is probably "too much". Lenses, in most of the incantations I've seen so far, are probably "too much". I'd love to see your solution for making
a.copy(b = a.b.copy(c = a.b.c.copy(d = a.b.c.d.copy(e = 1))))
As easy as a.b.c.d.e = 1
Closest I've seen is QuickLens, which is promising. Still pretty new and experimental, but definitely something I could see myself using: modify(a)(_.b.c.d.e).setTo(1)
Still slightly clunky, but less clunky than the other implementations I've seen e.g. in MonocleOne of the most important things a language can do, IMO, is make basic operations easily expressible. Lensing is one of those things.
Because when dealing with time, you get a lot of non-determinism when modifying values in variables. This happens because the ordering of non-commutative operations is important, mutable things have identity because their "value" changes with the history of those operations and so with variables and mutable data-structures it gets very hard to reproduce a certain state or to reason about the sequence of events. So in case of errors, it's hard to reproduce a bug or to reason about what ordering you had. Note this isn't to say that operation ordering issues won't happen with immutable data-structures and values - it can happen of course, though having values and pure functions in that chain will make it easier to debug.
But then if you introduce concurrency by multi-threading in the mix, that's when the shit really hits the fan. Because then you have to guarantee a certain ordering and you do that by concurrency primitives, most commonly by intrinsic locks and for one dealing with this stuff is really hard and non-intuitive and also locks are not composable and in general a bottleneck (Amdahl's law ftw). This is why people end up using `Future` and actors and other higher level abstractions for dealing with concurrency, because they make it easier to reason about ordering. But for those that ever worked with Erlang, the "receive" function of actors is meant to be pure, instead of modifying mutable variables, like people normally do in Akka. And in Akka you can work like in Erlang (e.g. context.become), but people rarely do that. And of course, you then get the most horrible mutation known in the software industry, because in addition to having shared mutable state, you're now also dealing with asynchronous message passing on top of that.
As a final word: having the ability to mutate is having power and when mutating variables you're using that power. Sometimes it's necessary, sometimes it's convenient, but many times it's more power than you need. You don't get the luxury of mutability when dealing with accounting for example ;-)
it's precisely when values change
over time that you want immutability
most.
Values always change over time, otherwise you could replace variables by constants.I suggest that immutability is most useful for data that crosses into a context that you (the programmer) have no/little control over. There is nothing wrong with a bit of state, like so
def pow ( n : Int ) : Int = {
var result = 1
var i = n
while ( i > 0 ) {
result *= 2
i -= 1 }
return result }
The reason this is OK is that the state changes are local, while globally (from the outside) pow remains functional. There is no need to coordinate with other programmers, or other functions or modules, to get i and n to behave properly.In contrast, if you pass shared state into a context that you have limited knowledge or control over (e.g. shared memory concurrency) then things become hairy.
That's why I advocate: "locally stateful, globally stateless" programming.
@tailrec
def pow(n: Int, result: Int = 1): Int =
if (n > 0) pow(n - 1, result * 2)
else result
But yes, mutability isn't problematic if it's well encapsulated. Works well for simple functions, but when you get to more complicated things (e.g. classes/modules) the problem is that proper encapsulation is very hard to do, sometimes next to impossible.Some people find some algorithms can be more naturally expressed using state. (And "some people" is a polite way of saying "alot".) The question of whether state can always be replaced by pure functions without loss of asymptotic complexity is open as of April 2016. All I wanted to point out is that there is no need of becoming a functional Taliban: if state is kept local, that's usually fine.
when you get to more complicated things
Yes. Then don't use it. My rule of thumb is: definition and all uses of state should be visible to the programmer without scrolling.You could try learning from a book or a course, but personally I've never found a better way of learning "what I need to learn for X" than trying to do X and then googling all the things I don't know.
Sure, you'll forget a whole bunch of that stuff, but if it's at all important you'll find yourself re-learning it again and again until it sticks.
Since you're here, on another subject, I've been toying with using Scala.js with Duktape to integrate Scala into a C++ project as a scripting language. Have you seen or heard anyone who has successfully done this or any pointers you may have?
p.s. Thank you for all you've contributed to the Scala community. Your posts are always enlightening and your projects incredibly forward thinking!
Great write-up regarding good software engineering practices in general, and ones applicable to Scala development specifically.
Two things I submit to be clarified/added to Strategic Scala Style are:
1 - Explicitly declare the return type of all public methods.
Doing so ensures that type inference does not propagate implementation types to the caller (often eliminating compilation errors) and, in my experience, greatly assists in reduction of errors when working with higher-kinded types. IMHO, it also assists in understanding a system due to an explicit contract being expressed.
2 - As "Joe Clueless" wrote in the comments, why isn't Either presented as a viable alternative to Option?
Again, IMHO, while Option is an exceptionally useful type, when indicating errors at the "public API level" it proves to be insufficient. The distjoint union type Either allows a collaboration to be both explicit that a failure may occur as well as indicate what the failure was (if any).
Hopefully these comments help and, again, kudos to a great write-up.
+1
This is very helpful for quickly scanning code.
>Again, IMHO, while Option is an exceptionally useful type, when indicating errors at the "public API level" it proves to be insufficient.
Agreed. Either, or even better, \/ (Scalaz either/disjunction) is much easier to work with and more expressive.
> Agreed. Either, or even better, \/ (Scalaz either/disjunction) is much easier to work with and more expressive.
I, too, recommend using the Scalaz \/[0] Either type as it is "right leaning" and lends itself quite nicely to processing due to it being both a model of Bifunctor[1] (making handling either a success or failure situation quite clean) and contributing nicely to defining an EitherT monad transformer capable of abstracting both latency (the Future part) and errors (the \/ part).
For example, I often have the following defined in projects:
type FutureEither[+T] = EitherT[Future, DomainError, T]
Where "DomainError" is a top-level type representing an error encountered during processing.All this adds up to being able to work with "happy path" logic, deferring error handling until it can be handled/reported and network request/response latency absorbed until it becomes an error (addressed as any other error would be).
0 - https://github.com/scalaz/scalaz/blob/series/7.3.x/core/src/...
1 - https://github.com/scalaz/scalaz/blob/series/7.3.x/core/src/...
As to Either, I didn't mention it because I don't use it much, and don't see many other people using it either. Sure you can use it and it'll work, but it's not the style I use or the style I see most open source libraries using.
As for Scalaz, yeah \/ is great, or cats' Xor. But this doc is all about Vanilla Scala: for every library out there, they will have their own best practices, that are totally uninteresting to people not using hat library. If someone wants to write similar documents for Scalaz, Cats, Akka, Play, Finagle, Play, they would each probably be just as long as this one, and probably just as valuable!
IMHO your scope is spot on for Vanilla Scala and will help many.
Regarding the similar documents shout-out, here's one for Scalaz which I found very well written:
http://eed3si9n.com/learning-scalaz/index.html
Perhaps others know of write-ups similar to yours and this one for some of the others you mentioned. Could be interesting to compile them into a "see also" kind of thing.
What I'd really like is a simple type union syntax, so I could union the happy path result with any number of failure modes, where there isn't necessarily a common supertype for anything.
I frequently see Either abused to represent the lack of Union Types. But then I also see it used for errors. And it's not really a perfect fit for either (no pun intended).
Every (a non-Empty List) is also nice. I like Scalactic. It adds just a couple of small useful things. Very lightweight.
Option (Some or None), Or (Good or Bad) and Try (Success or Failure[T <: Throwable]) cover the bases for me.
Just in case others were unaware of scalactic. (sister library to scalatest, so if you're already using scalatest, you're already using scalactic under the covers I believe)
If all you need is represent an optional value, why use `Either`, which needs a second type parameter representing the failure? If you don't care about what caused the failure (because there was no failure, an absence of value is a valid result), no need to represent it.
Besides, these two classes still represent fundamentally different concepts and they are not interchangeable at all.
Sorry for any confusion I may have introduced when re-stating the question originally posed by "Joe Clueless" in the comments section.
- If you want to extract them out, it's safe and straightforward. Still a tedious refactor, but a safe tedious refactor. It's not any harder than injecting stuff up-front. Given that, why do the work now if it's not any harder to do later, and you may not need to do it?
- If you hardcode dependencies except for those you can't hardcode, that means that if you come across something which isn't hard coded, you know immediately there must be some reason it needs to be injected in! This makes reading code easier, since following this convention the code itself tells you how it's used, vs having to guess whether an injected dependency is actually necessary or whether it has just one instantiation and can be hardcoded
- You are already hardcoding all sorta of things anyway. The `Seq` object? The `String` type? These, and other things, can be injected in as params or type-params if we really want to. The question is then when do you stop and draw the line? You obviously can't inject everything. "Only inject stuff that needs to be injected" provides a clear, unambiguous guideline that applies equally to dependencies on your code, on the std lib, or on third party libraries
Thanks for your input!
Related in that you mentioned dependency injection, unrelated in that it is not directly relevant to the content of Strategic Scala Style. (maybe of interest to some though).
0 - https://yow.eventer.com/yow-2012-1012/lambda-the-ultimate-de...
Disclaimer: just a fan of the book :-)
Putting on my management hat that thought fills me with horror.
For example pull requests become a nightmare when you have extremely experienced Scala devs using every trick in the book and producing code which no one else on the team understands except those with similar levels of Scala knowledge.
Honestly, it's greek for the rest of us and furthermore it seems each Scala expert has their own way of implementing the same functionality.
I enjoy programming in Scala, I write code in it (when I have to) but in all honesty as cool as it is we find ourselves needing to stick with Python, Java and JavaScript to ensure we have some sort of consistency between new hires and experienced devs.
That's why this document exists at all. We can definitely make things better, but to do so requires putting together guidelines, proposals, and consensus (things developers are historically pretty poor at), rather than just cranking out code and complaining (both of which developers excel at, especially fueled by coffee)
Then no one would ever use scala?
For me the reasons I use Scala rather than Java/Python/etc. are the same reasons I would use Java/Python/etc. rather than assembly. To write code with the same guarantees as the core of my system would be theoretically possible in a "simpler" language, but it would be incredibly verbose: it would take longer to write and be much harder to maintain (because you'd have to keep a lot more in your head to understand each piece of functionality). If I weakened the guarantees (i.e. didn't keep track of as many possible side effects) I'd have a higher defect rate, especially when refactoring.
That's the small heart of a few projects, but I see a lot of merit in using the same language for your whole codebase: http://www.teamten.com/lawrence/writings/java-for-everything... . And Scala is actually a pretty nice language for one-off throwaway scripts where you don't care about tracking any effects or similar as well.
I could use the same reasoning to justify writing my code in Common Lisp instead. The thing about Turing-complete languages is that the only real dimmension of power is their expressiveness - i.e. how much boilerplate do you have to write to implement something (and therefore how much of that cruft other people have to deal with when trying to understand your code).
I therefore find this principle as applied to programming languages to be illogical.
It makes sense in non-Turing-complete languages though (configurations, DSLs) - because there by "least power" one means power in the computational sense. Accidentally making a language too powerful when it's not needed can lead to security issues.
Maybe you should then. I find it too hard to restrict my code in lisps: if I want a certain part of my code to be in a DSL-like constrained sublanguage, there's no standardized way to express that, only ad-hoc macros.
> It makes sense in non-Turing-complete languages though (configurations, DSLs) - because there by "least power" one means power in the computational sense.
I think non-Turing-complete general-purpose languages are possible. I'm very excited for Idris.
Unfortunately, Scala's typesystem is turing complete.
There is a standardized way - via "ad-hoc" macros. Except they're not really more "ad-hoc" than using classes and traits in Scala is "ad-hoc".
Maybe 'aninhumer has a point - restricting the language is kind of the opposite goal of making it expressive/powerful.
> I think non-Turing-complete general-purpose languages are possible. I'm very excited for Idris.
Idris is not Turing-complete? :o.
My point was more that safety is a separate, important metric of language quality. It often conflicts with expressiveness, but not necessarily.
See also SPJ's "Haskell is Useless" video: https://www.youtube.com/watch?v=iSmkqocn0oQ
They're not standardized enough to have good common tooling around restricted embedded DSLs. If you use the type system to enforce restrictions around certain areas of Scala code, every Scala tool understands it. IME a lot of lisp programmers end up writing their own tool integration, because there is no standard for that kind of restriction (macros can be arbitrary code; in practice programmers restrict themselves to sensible macros, but in a way that isn't exposed to tooling).
> Idris is not Turing-complete? :o.
In practice there are escape hatches, but it's a total language: your function must come with a proof that it terminates.
So, you can't express the programs that never terminate, but can you express all the programs that would terminate?
Point in case, you probably shouldn't ever want to use a language for the specific feature that could express a non-deterministic program.
This gets philosophical - would the program "Y {f => x => if(x is a proof of the inconsistency of PA via the Goedel encoding) 0 else f(x+1) } 0" terminate? If you assume consistency of PA then no.
> Point in case, you probably shouldn't ever want to use a language for the specific feature that could express a non-deterministic program.
Well the mu operator or equivalent (informally, the inverse operator) is an extremely useful feature. Many useful programs can't be written strictly primitive recursively (as a trivial example, you can't compute the Ackermann function). I have hope that we can find more restricted versions of mu that let us express all the programs we want to, but that's a decidedly nontrivial problem.
Have fun implementing your compile-time Common Lisp typechecker.
I mean, you can do it. I have friends implementing a typechecker for Python, and it works. As for me, I'm very happy to have someone else implementing/debugging/maintaining a typechecker for me so i can do other things with my time, like watch cats climb into boxes on youtube.
But anyway, that's kind of my point. Typechecker is the kind of excess feature the principle of least power would like you to reject.
For example, Kotlin doesn't support higher kinded types so if your problem doesn't require highed kinded types, Kotlin would be a better fit than Scala since it's the "Least powerful language that will solve your problem".
Which will probably lead to the controversial claim that "All programs need higher kinds" followed by the realization that the perceived power of a technology is extremely subjective.
> When defining an interface to a package someone else will use...
> 1. The simplest interface to a package is static method with standard types
> 2. Next, static methods that take/return custom types, presumably with methods
> 3. Next, is needing the user to instantiate some classes
> 4. Lastly, is needing the user to inherit from some classes