Slides of my Scala Lift Off 2011 talk. The content of the presentation is mostly similar to the one presented at Scala Days 2011, with a few additions. Particularly, lazy values are discussed.
Managing Binary Compatibility in Scala (Scala Lift Off 2011)
1. Managing Binary Compatibility in Scala
Mirco Dotta
Typesafe
October 13, 2011
Mirco Dotta Managing Binary Compatibility in Scala
2. Introduction Sources of Incompatibility Conclusion
Outline
Introduction
Example
Scala vs. Java
Sources of Incompatibility
Type Inferencer
Type Parameters
Trait
Lazy Fields
Conclusion
Mirco Dotta Managing Binary Compatibility in Scala
3. Introduction Sources of Incompatibility Conclusion Example Scala vs. Java
Example
Assume Analyzer is part of a library we produce. We decide that
its API has to evolve as follows:
class Analyzer { // old version class Analyzer { // new version
def analyze(issues: HashMap[ , ]) {...} def analyze(issues: Map[ , ]) {...}
} }
Further, assume the next expression was compiled against the old
library
new Analyzer().analyze(new HashMap[Any,Any])
Would the compiled code work if run against the new library?
The answer lies in the bytecode...
Mirco Dotta Managing Binary Compatibility in Scala
4. Introduction Sources of Incompatibility Conclusion Example Scala vs. Java
Example: Bytecode
Let’s have a look at the method signature for the two versions:
class Analyzer { // old version class Analyzer { // new version
analyze(Lscala/collection/immutable/HashMap);V analyze(Lscala/collection/immutable/Map);V
} }
The expression compiled against the old library would look like:
...
invokevirtual #9;// #9 == Analyzer.analyze:(Lscala/collection/immutable/HashMap;)V
⇒ The method’s name has been statically resolved at
compile-time.
Running it against the new library would result in the JVM
throwing a NoSuchMethodException.
⇒ The evolution of class Analyzer breaks compatibility with
pre-existing binaries.
Mirco Dotta Managing Binary Compatibility in Scala
5. Introduction Sources of Incompatibility Conclusion Example Scala vs. Java
Is Binary Compatibility a Scala issue?
The short answer is No. The discussed example can be easily
ported in Java or other languages targeting the JVM.
Scala shares with Java many sources of binary incompatibility.
But Scala offers many language features not available in Java:
First-class functions.
Type Inferencer.
Multiple inheritance via mixin composition (i.e., traits).
Lazy values.
. . . Just to cite a few.
⇒ Scala code has new “unique” sources of binary incompatibility.
Mirco Dotta Managing Binary Compatibility in Scala
6. Introduction Sources of Incompatibility Conclusion Type Inferencer Type Parameters Trait Lazy Fields
Type Inferencer: Member Signature
Does the following evolution break binary compatibility?
class TextAnalyzer { // old version class TextAnalyzer { // new version
def analyze(text: String) = { def analyze(text: String) = {
val issues = Map[String,Any]() val issues = new HashMap[String,Any]
// ... // ...
issues issues
}} }}
Question
What is the inferred return type of analyze?
Let’s compare the two methods’ signature.
class TextAnalyzer { // old version class TextAnalyzer { // new version
public scala.collection.immutable.Map public scala.collection.immutable.HashMap
analyze(java.lang.String); analyze(java.lang.String);
} }
Mirco Dotta Managing Binary Compatibility in Scala
7. Introduction Sources of Incompatibility Conclusion Type Inferencer Type Parameters Trait Lazy Fields
Type Inferencer: Member Signature (2)
Question
Can we prevent the method’s signature change?
That’s easy! The method’s return type has to be explicitly
declared:
class TextAnalyzer { // bytecode compatible new version
def analyze(text: String): Map[String,Any] = {
val issues = new HashMap[String,Any]
// ...
issues
}}
Take Home Message
Always declare the member’s type. If you don’t, you may
inadvertently change the members’ signature.
Mirco Dotta Managing Binary Compatibility in Scala
8. Introduction Sources of Incompatibility Conclusion Type Inferencer Type Parameters Trait Lazy Fields
Type Parameters & Type Erasure
Assume NodeImpl <: Node. Does the following evolution break
binary compatibility?
class Tree[T <: NodeImpl] { // old version class Tree[T <: Node] { // new version
def contains(e: T): Boolean = {...} def contains(e: T): Boolean = {...}
} }
Type parameters are erased during compilation, but the erasure of
Tree.contains is different for the two declarations.
boolean contains(NodeImpl) {...} boolean contains(Node) {...}
Remember that the JVM looks up methods based on the textual
representation of the signature, along with the method name.
Take Home Message
Type parameters may change the members’ signature and hence
break binary compatibility.
Mirco Dotta Managing Binary Compatibility in Scala
9. Introduction Sources of Incompatibility Conclusion Type Inferencer Type Parameters Trait Lazy Fields
Trait Compilation
Traits are a powerful language construct that enables
multiple-inheritance on top of a runtime – the JVM – that does
not natively support it.
Understanding how traits are compiled is crucial if you need to
ensure release-to-release binary compatibility.
So, how does the Scala compiler generate the bytecode of a trait?
There are two key elements:
A trait is compiled into an interface plus an abstract class
containing only static methods.
“Forwarder” methods are injected in classes inheriting traits.
Mirco Dotta Managing Binary Compatibility in Scala
10. Introduction Sources of Incompatibility Conclusion Type Inferencer Type Parameters Trait Lazy Fields
Trait Compilation Explained
An example will help visualize how traits get compiled:
// declared in a library
trait TypeAnalyzer {
def analyze(prog: Program) { // client code
// the trait’s method impl code class TypingPhase extends TypeAnalyzer
}
}
The following is the (pseudo-)bytecode generated by scalac:
interface TypeAnalyzer { class TypingPhase implements TraitAnalyzer {
void analyze(Program prog); // forwarder method injected by scalac
} void analyze(Program prog) {
abstract class TypeAnalyzer$class { // delegates to implementation
static void analyze(TypeAnalyzer $this, TypeAnalyzer$class.analyze(this,prog)
Program prog) { }
// the trait’s method impl code }
}
}
Mirco Dotta Managing Binary Compatibility in Scala
11. Introduction Sources of Incompatibility Conclusion Type Inferencer Type Parameters Trait Lazy Fields
Trait: Adding a concrete method
Question
Can we add a member in a trait without breaking compatibility
with pre-existing binaries?
Mirco Dotta Managing Binary Compatibility in Scala
12. Introduction Sources of Incompatibility Conclusion Type Inferencer Type Parameters Trait Lazy Fields
Trait: Adding a concrete method
trait TypeAnalyzer { // new version // compiled against the old version
def analyze(prog: Program) {...} class TypingPhase implements TraitAnalyzer {
def analyze(clazz: ClassInfo) {...} // forwarder method injected by scalac
} void analyze(Program prog) {
// delegates to implementation
TypeAnalyzer$class.analyze(this,prog)
//TypeAnalyzer trait compiled }
interface TypeAnalyzer { // missing concrete implementation!
void analyze(Program prog); ??analyze(ClassInfo clazz)??
void analyze(ClassInfo clazz); }
}
abstract class TypeAnalyzer$class {
static void analyze(TypeAnalyzer $this, Take Home Message
Program prog) {...}
static void analyze(TypeAnalyzer $this, Adding a concrete method in a trait
ClassInfo clazz) {...} breaks binary compatibility.
}
Mirco Dotta Managing Binary Compatibility in Scala
13. Introduction Sources of Incompatibility Conclusion Type Inferencer Type Parameters Trait Lazy Fields
Lazy values decompiled
Lazy values are a Scala language feature that has no
correspondent in the JVM.
⇒ Their semantic has to be encoded in terms of the available
JVM’s primitives.
A lazy val is encoded into a private field and a getter plus a
bitmap.
The bitmap is used by the getter for ensuring that the private
field gets initialized only once.
The bitmap is of type Int, meaning that a maximum of 32
lazy value can be correctly handled by it.
The bitmap is shared with subclasses, so that a new bitmap
field is created only if strictly necessary.
Mirco Dotta Managing Binary Compatibility in Scala
14. Introduction Sources of Incompatibility Conclusion Type Inferencer Type Parameters Trait Lazy Fields
Lazy to Eager
Question
Can we transform a lazy field into an eager one?
Mirco Dotta Managing Binary Compatibility in Scala
15. Introduction Sources of Incompatibility Conclusion Type Inferencer Type Parameters Trait Lazy Fields
Lazy to Eager
Let’s take a simple example and look at the generated bytecode:
class ClassAnalyzer { // old version class ClassAnalyzer { // new version
lazy val superclasses: List[Class] = { val superclasses: List[Class] = {
// ... // ...
} }
} }
And here is the bytecode:
class ClassAnalyzer { // old version
class ClassAnalyzer { // new version
private List[Class] superclasses;
private final List[Class] superclasses;
volatile int bitmap$0
public int superclasses();
public int superclasses();
// ...
// ...
}
}
Take Home Message
Transforming a lazy value into an eager one does preserve
compatibility with pre-existing binaries.
Mirco Dotta Managing Binary Compatibility in Scala
16. Introduction Sources of Incompatibility Conclusion Type Inferencer Type Parameters Trait Lazy Fields
Eager to Lazy
Question
Can we transform an eager field into a lazy one?
Mirco Dotta Managing Binary Compatibility in Scala
17. Introduction Sources of Incompatibility Conclusion Type Inferencer Type Parameters Trait Lazy Fields
Eager to Lazy
The previous example demonstrates that lazy and eager fields are
syntactically equivalent, from a client perspective.
Does that mean that the transformation is safe?
It depends.
If you are in a closed world, then the transformation is safe
(e.g., your class is final).
Otherwise, you are taking a high risk on breaking the semantic
of lazy, and no good can come out of that. Remember that
the bitmap is shared between subclasses.
Take Home Message
Transforming an eager value into a lazy one may not preserve
compatibility with pre-existing binaries.
Mirco Dotta Managing Binary Compatibility in Scala
18. Introduction Sources of Incompatibility Conclusion Conclusion Future Work Scala Migration Manager
Conclusion
Ensuring release-to-release binary compatibility of Scala
libraries is possible.
Though, sometimes it can be difficult to tell if a change in the
API of a class/trait will break pre-existing binaries.
In the discussed examples we have seen that:
Type inferencer may be at the root of changes in the member’s
signature.
Type parameters may also modify members’ signature.
Traits are a sensible source of binary incompatibilities.
Transforming an eager field into a lazy one can break semantic.
It really looks like library’s maintainers’ life ain’t that easy...
Mirco Dotta Managing Binary Compatibility in Scala
19. Introduction Sources of Incompatibility Conclusion Conclusion Future Work Scala Migration Manager
Scala Migration Manager (MiMa)
A few months ago we released the Scala Migration Manager!
It’s free!!
It can tell you, library maintainers, if your next release is
binary compatible with the current one.
It can tell you, libraries users, if two releases of a library are
binary compatible.
MiMa can collect and report all sources of “syntactic” binary
incompatibilities between two releases of a same library.
“Syntactic” means NO LinkageError (e.g.,
NoSuchMethodException) will ever be thrown at runtime.
Now, it’s time for a short demo!
Mirco Dotta Managing Binary Compatibility in Scala
20. Introduction Sources of Incompatibility Conclusion Conclusion Future Work Scala Migration Manager
Future Work
Reporting binary incompatibilities is only half of the story. We are
working on a “companion” tool that will help you migrate binary
incompatibilities.
For the reporting there are many ideas spinning around. Your
feedback will help us decide what brings you immediate value
One that I believe is useful:
Maven/Sbt integration.
Mirco Dotta Managing Binary Compatibility in Scala
21. Introduction Sources of Incompatibility Conclusion Conclusion Future Work Scala Migration Manager
Scala Migration Manager
Visit http://typesafe.com/technology/migration-manager
More information about the Migration Manager
Download it and try it out, it’s free!
We want to hear back from you.
Success stories
Request new features
Report bugs
Want to know more, make sure to get in touch!
email: mirco.dotta@typesafe.com, twitter: @mircodotta
Mirco Dotta Managing Binary Compatibility in Scala
22. Introduction Sources of Incompatibility Conclusion Conclusion Future Work Scala Migration Manager
One last thing I didn’t mention...
We will make the sources public!
Mirco Dotta Managing Binary Compatibility in Scala