This is so awesome! I've been slowly learning Scala and was looking for a "Effective Scala" type book.
Thanks for making this!
Thanks for making this!
Much of it will depend on your application domain and needs. For example, if you're dealing with matrices, avoiding destructive updates will only give you a performance hit without an increase in maintainability.
Sorry of for the simple question, but I think I'm missing something. Wouldn't avoiding a destructive update give better performance? And the trade-off would be higher maintainability? With a mutable structure, you're 'destroying' just reusing the same locations in memory and just replacing, versus say an immutable structure returning a completely new structure (new memory allocations).
But to your main point, yes I won't take this as the canonical source of coding standards. I just like to read through stuff like this because it gives me alot of areas to explore and research. Like why do this vs that, what's the tradeoffs, etc..
<<edit>> Whoop's looks like i have my logic/terminology flipped! Saw this on SO which cleared things up a bit.
http://stackoverflow.com/questions/6964233/what-is-a-destruc...
Also just noticed the + vs += in some of the mutable collections, which helps to explain.
Much of section 2 and 4. Also, I'm not using Play, Akka, or any other web frameworks (since I don't do web stuff), so those don't apply to me and I don't have a basis for evaluating them.
> Wouldn't avoiding a destructive update give better performance?
No. In the absence of destructive updates, you either need to copy matrices or use a data structure that typically incurs a log(n) overhead with a fairly large constant.
That is not true - first of all, log(n) in our industry means log2(n). Scala's Vector for example is a log32(n), which given that the max size is Int.MaxValue, can be treated as constant access as far as algorithmic complexity is concerned. Persistent HashMaps are also implemented (instead of self-balancing binary trees), which again means less than log2(n). And Scala's Queue for example is O(1) for both enqueue() and dequeue() - it's just that every "n" operations it has to invert a list of pending items, which takes "O(n)" but given that it happens every "n" operations it is amortized.
It's not algorithmic complexity that bites you, algorithmic complexity is totally fine. The real problem is more low level, with the indirections happening due to usage of pointers. Depending on memory usage patterns, persistent data-structures could stress the GC enough to increase the number of stop-the-world pauses. Also in a multi-threading context, when you keep the pounding on the same reference holding an immutable data-structure, you drive the contention to the root of that tree and that is less efficient than a specialized mutable data-structure that can distribute those locks in multiple buckets ... but still far better than synchronizing on a standard mutable data-structure.
Actually, which is better in terms of performance depends on context. For example if you have shared reads, then an immutable data-structure is better because you do not need to synchronize those reads.
Also - I did work on a web service that was hit with tens of thousands of requests per second and because of the architecture we've built (asynchronous single producers, multi consumers), usage of immutable data-structures wasn't an issue and it was much saner. AND most people working on most project never end up with such hard optimization issues.
Therefore fear-mongering about something that in practice really is not an issue is not helpful, especially given the sanity that usage of immutable data-structures brings. See this other rule I added - https://github.com/alexandru/scala-best-practices/blob/maste...
Which, for a 1000x1000 matrix, means a factor of about 20, not to mention the significant constant overhead from not using a contiguous memory layout.
> It's not algorithmic complexity that bites you, algorithmic complexity is totally fine.
I didn't say anything about algorithmic complexity. Note that I was writing log(n), not O(log(n)) in particular and pointing at the constant factor, too. A computation that takes two days instead of half an hour does make a very real difference to me.
> I did work on a web service that [...]
That's why I was careful to point out how different application domains matter. Web services are not numerical computations are not computer algebra are not big data.
IMHO, such a big difference only arrises when big differences in algorithmic complexity is involved.
Again I must mention that I'm of the opinion that if you know what you're doing, then it's fine, but you must be able to defend your choice, as an argument like "gosh, heard from the Internet that immutable data-structures are slower" is not acceptable given the extra sanity that it brings. Plus, depending on context (e.g. web services that are dealing with parallelism combined with I/O) immutable data-structures might actually improve throughput.
Your use-case is entirely legitimate, therefore that rule on usage of immutable data-structures is a SHOULD (optional), rather than a MUST (required). If you wanna do it, then do it - after all, the ability to get down and dirty is one thing I like about Scala, too bad that it doesn't have a GC-less version :-)
No, this arises as the result of large constant factors, too. As I pointed out, log2(1e6) is approximately a factor of 20 alone and you pay additional constant overhead for having suboptimal memory layout and allocations. Remember, while functional programming languages love trees and linked lists, the hardware loves contiguous arrays. This is what the L1 and L2 cache and prefetching logic of your typical CPU are optimized for.
I'm not against destructive updates, but I'm against doing destructive updates without knowing what you're doing. I'm also against usage of shared state without proper encapsulation and synchronization.
And when a developer makes such a problematic choice, it has to be able to defend it with arguments other than "couldn't think of anything better and we've got deadlines". I hate that argument with a passion, time is somehow always used as an argument for doing stupid shit, technical debt is then increased and then sprint after sprint you're left cleaning up stupid shit from previous sprints.
Yes, I'm very opinionated :-)
With respect to your points:
> I'm not against destructive updates, but I'm against doing destructive updates without knowing what you're doing.
With all due respect, the document doesn't make this clear. See, e.g.: "a public API exposing a mutable data-structure is an abomination of nature". That's a pretty absolute statement.
> I'm also against usage of shared state without proper encapsulation and synchronization.
There are really two parts to this. First of all, in principle I agree with the above, but that's not what your document says. For example, section 4.1 goes much farther than that; it may be exaggerated overstatement for effect, but it's really not backed up by anything.
Admittedly, part of the problem is that Scala has never fixed Java's broken [1] concurrency model (and may not be able to do so due to interoperability issues), but even so, this is a statement that's difficult to support in the general case.
There are also sections that don't really make sense, to be honest. For example, 4.7: Not only isn't there really a formal definition of "thread-safe", any attempt to define it (such as using non-interference per Owicki/Gries) makes it essentially a global property of the program, not the local property of a module (let alone its API). There are software design and implementation techniques that allow you to have composable module specifications in the face of concurrency, but that's something different.
In a way, yes, avoiding concurrency is not unreasonable advice in the face of the limited tools you have available and you can say that I'm lamenting more the (lack of) actual support for concurrency in modern programming languages rather than criticizing your statement; at the same time, even given the circumstances, section 4 strikes me as overall too limiting. There ARE other techniques to handle concurrency cleanly, after all.
[1] I use the word "broken" advisedly. See http://brinch-hansen.net/papers/1999b.pdf
On 2.9/2.14, as somebody that suffered from servers crashing due to enormous load and that had to profile the shit out of everything, I usually find such arguments against persistent data structures to be bullshit, and if you think it matters that much for your problems, you probably picked the wrong language and possibly the wrong platform. And NO, there is no conflict with 3.4.
On destructive updates again, there is a fine line between interesting and boring, plus I don't really have time to write a book. But thanks for the feedback, I'll add more details.
On 4.1, I really hope you're making a difference between concurrency and parallelism, right?
I don't need to back that claim up, just as I don't need to prove that the sky is blue. And as I was telling somebody else, concurrency is not something to take advantage of, but rather something one has to suffer from.
On Java's "broken concurrency model", actually I find Java to be quite sane, because at least it has a memory model that works well cross platform and that is generic enough to support all kinds of abstractions built on top, like a sane Future with C#-like async, or light weight threads (Quasar), or extremely efficient queues (LMAX Disruptor), or Erlang-style Actors (Akka), or reactive streams (Rx), or parallel collections (Scala), or STM that works (Scala-STM) or CSP or Agents or what have you. And because there is no silver bullet, all of these abstractions are useful to have in your toolbox. And surely a platform built for a certain abstraction (eg Erlang) will be better than a library, but also find that to be very limiting.
Again I challenge you to give me an example of a platform that handles concurrency better than Java.
Related to my list, I wasn't the one that posted it BTW, not trying to impose my will on anybody, take from it what you will.