ScalaDays 2012 – Odersky's Keynote: Where Scala is Going

http://skillsmatter.com/podcast/home/scala-days-keynote-3018

Retrospective (00:00)

- First ScalaDays in Lausanne (2010), sold out with 150 attendees. Scala 2.8 was announced.

- Second ScalaDays in Stanford (2011), sold out with 260 attendees. Commercial support, Typesafe Stack with Scala 2.9 and Akka 1.1.

- Third ScalaDays in London (2012), sould out with 400 attendees. Typesafe Stack with Scala 2.9.2, Akka 2.0, Play! 2.

Where we are now (05:45)

- Scala 2.10 on the horizon, basically feature-complete.

- Refinements, testing and release, will happen in the next months.

Scala 2.10 overview (06:20)

- Lots of work on tooling: Eclipse, IntelliJ, ... more features, better performance.

- Low-level concurrency framework in the standard library (Futures and Promises), used by both parallel collections and Akka.

- Native reflection framework.

- A few new language features.

Reflection Framework overview (08:05)

- Cake pattern has a central role, improvements in 2.10 (dependent method types) make it more useful.

- Compilers and reflection are quite similar: Both need to deal with the same concepts, answer similar questions.

- Differences between compilers and reflection: Where does the information come from?, Generate code vs. invoke pre-generated code, error messages vs. exceptions, thread-safety, (im)mutability of types, ...

Reflection in 2.10, demonstration (12:05)

Mirror-based reflection (16:50)

- Mirrors return the reflective information of runtime values.

- Different mirrors can provide different characteristics.

- Types of different mirrors are enforced to be incompatible with each other: Reflects the design of VMs that for instance even the same types on different machines are distinct.

- Common supertype allows reasoning about all mirrors.

Implementation of reflection (18:30)

- Functionality for trees, symbols, types, names.

- Functionality to decompose and explore relationships between them, ...

- Roughly equivalent to the core of the language specification and the core of the compiler.

Comparison with Java (19:15)

- Java has a quite complex typesystem since Generics were introduced.

- What is Java approach? Pretty much empty interfaces, no useful methods defined.

- Want to do anything non-trivial with types in Java? Write your own compiler/typechecker!

- Why was it done this way? Would have required to write essential parts of a compiler (hard) and to make sure both compilers always agree (almost impossible). Smart decision of the Java designers to not suffer this maintenance nightmare.

Scala's approach (22:00)

- Unify the core parts of compilers and reflection.

- Compiler architecture: Object-oriented encapsulation, seperation of concerns. Use a cake pattern in which every component has a self-type that contains all its required dependencies ("Slices of the cake").

- How to integrate reflection?

- Multiple cakes: compiler cake (`scala.nsc.Global`) and reflection cake (`scala.reflect.runtime.Mirror`) inherit from common super-cake (`scala.reflect.internal.Universe`), which captures common information.

- Refactor common functionality into super-cake, let individual cakes handle differences (e. g. compiler cake: IO, reflection cake: thread synchronisation)

- Refine super-cake (`scala.reflect.internal.Universe`) by providing a cleaned-up facade to the user on top, `reflect.api.Universe`.

- Java's interfaces not enough, Scala leverages abstract types to both present a convenient interface to the user and allow elegant/efficient implementation behind the scenes.

Reflection summary (29:50)

- Scala is a pretty regular language:

- Everything can be nested: Classes, methods, objects, types.

- Everything can be abstract: Methods, values, types.

- Type of `this` can be declared freely to express dependencies.

- Enables expression of cake hierarchies for software design in the large.

Macros overview (30:50)

- Once reflection and compilers are unified – what can we do now?

- What happens when the compiler itself can make use of reflection?

- It can call user-defined methods during compilation, which can consume/produce trees/types.

- This leads to a simple macro system, which is included as an experimental feature in Scala 2.10.

Macro definition (32:40)

- Standard Scala methods declaration with all the same rules for overloading/overrididng/... + implementation part which points to a concrete, stable macro implementation. Both have standard Scala method signatures.

- Types are lifted from `T` to `Expr[T]`, which can be descriped as an AST which, when run, produces a result of type `T`.

Macro expansion (34:35)

- If the compiler encounters a macro application during type-checking, it will first type-check the arguments (as usual), expand the macro application by doing a reflective call to the macro implementation with the context (call site info) and the AST. Then the macro implementation runs and produces a tree, which will be taken as a replacement of the macro application and type-checked again.

Macro example (35:55)

Expressing syntax trees (37:00)

- Scala provides no special/new syntax. Writing down syntrax trees might be cumbersome, but there is actually a better way to do it without introducing special syntax.

Contexts (40:00)

- A macro context contains a mirror (in this case the current compiler instance) that anchors trees, types which are being passed in and out of the macro.

- Also contains information about the macro call.

Hygiene (41:25)

- Lot of hard work in the Scheme community in the 80ies to avoid hygiene issues.

- Scala's macro system is trivial, does nothing more than accepting trees/types and producing trees again, not hygienic. "Not so good."

- Real motivation for macros was `reify`: Takes expression of type `T` and produces a tree representing this expression of type `Expr[T]` at runtime.

- For example to create a query from filter/map expressions.

- See also LINQ on .NET.

- `reify` can be (and is) implemented as a macro, enables composition.

- `reify` and `eval` are inverses, `reify: T => Expr[T]`/`eval; Expr[T] => T`.

- `reify`( and `eval`) enable:

- Splicing.

- Hygiene enforcement, types will prevent mistakes.

- No need for special syntax for syntax trees.

Macro summary (51:55)

- Very nice syntax and hygienic macros without much effort.

- Cost/benefit ratio is huge.

- Bootstrap operation: Minimal, unhygienic macro system -> Reification as a macro -> Code-as-syntax-tree + hygiene.

- Macros are a better solution than compiler plugins, because APIs are kept stable.

Language features overview (54:00)

- Scala Improvement Process (SIP) with a lot of community involvement.

- SIP11: String interpolation

- SIP13: Implicit classes

- SIP14: Futures and Promises

- SIP15: Value classes

- SIP17: Type Dynamic

- SIP18: Language modularity

- Small additions, but interesting implications.

SIP11: String interpolation (54:50)

- Constructing strings with `+` is very cumbesome.

- Embedded strings would be rather nice, but it is not possible to just reinterpret `$`, because it is already a legal symbol inside a string.

- Syntax is itself extensible: `id"..."` gets translated to `StringContext("...").id(<args>)`.

- Enables format strings, embedded XML, JSON, SQL, Scala, JavaScript, ... just provide an appropriate method on `StringContext`.

SIP13: Implicit classes (59:50)

- "Everyone got extension methods! Why doesn't Scala?" – "Actually you want implicits, because implicits can not only add methods but also implement new interfaces, and in mature software systems methods exist for a purpose – to implement interfaces."

- Scala 2.10 completely eliminates syntactic and runtime overhead of implicits:

- Syntactic overhead, adressed by SIP13: Implicit classes

- Runtime overhead, addressed by SIP15: Value classes

- Risk of misuse, adressed by SIP18: Language modularity

- Before: `class Bar(foo: Foo) { ... }` + `implicit def fooToBar(foo: Foo) = new Bar(foo)`

- After: `implicit class Bar(foo: Foo) { ... }`

SIP15: Value classes (63:50)

- Drop restriction that user-defined classes have to (explicitly or implicitly) inherit `AnyRef` (`java.lang.Object` in Java).

- Scala classes can now also inherit from `AnyVal` (only `Int`, `Long`, etc. could inherit from `AnyVal` before).

- `AnyVal` classes have no object identity and only allow a single public value.

- Instances of those classes can be created and destroyed by the compile at will – basically as transparent as boxing/uunboxing in Scala before.

- Methods of `AnyVal` are compiled down to extension methods ("static" methods in Java class files).

- Combination of implicit and value classes results in both efficiency and conciseness of extension methods with all the advantages of implicits.

Summary (66:15)

- Scala has grown to an accepted and popular technology for scalable applications.

- Development continues at a rapid pace – both at the language and the middleware level.

- Invaluable community involvement in refining the language and its libraries.