Zinc talk given at ScalaSphere 2019

1. Analysis of Zinc
Eugene Yokota (@eed3si9n)
Lightbend

2. previously in ScalaSphere 2017...

3. separate compilation

4. separate compilation
A.scala
A.class B.scala
B.class

5. separate compilation
A.scala
A.jar B.scala
B.class

6. A.scala
A.class B.scala
B.class
A.sig
"Types are a property of expressions
and thus exist [only] at compile time"
— Creative Scala, Gurnell and Welsh

7. A.scala
A.class B.scala
B.class
A.sig
"I turned myself into a pickle, Morty!"
— Rick and Morty

8. A.scala
A.class B.scala
B.class
A.sig
def foo: Unit = {
val a = new A
a.something
}

9. A.scala
A.class B.scala
B.class
A.sig
def foo: Unit = {
val a = new A
a.something
}
scala> :javap example.B
...
public void foo();
descriptor: ()V
flags: ACC_PUBLIC
Code:
stack=2, locals=2, args_size=1
0: new #12 // class example/A
3: dup
4: invokespecial #15 // Method example/A."<init>"
7: astore_1
8: aload_1
9: invokevirtual #19 // Method example/A.somethin
12: pop
13: return

10. incremental
(re)compilation =
minimality

11. incremental
(re)compilation =
minimality
• compiles sources at most once per build
only if they transitively depend on
sources that changed since the previous
build

12. naïve incremental
(re)compilation
A.scala B.scala C1.scala
C3.scala
C2.scala
C4.scala
• step 1

13. naïve incremental
(re)compilation
• step 2
A.scala B.scala C1.scala
C3.scala
C2.scala
C4.scala
initial invalidation

14. naïve incremental
(re)compilation
• step 3
A.scala B.scala C1.scala
C3.scala
C2.scala
C4.scala
transitive invalidation

15. naïve incremental
(re)compilation
• step 3
A.scala B.scala C1.scala
C3.scala
C2.scala
C4.scala
problem: results to compiling
all/most sources in the graph

16. name hashing

17. name hashing
• created by Grzegorz Kossakowski in
2013 (sbt/sbt#1026)

18. name hashing
• we need to prevent invalidation from
spreading like a wildfire

19. name hashing
• step 1
A.scala B.scala C1.scala
C3.scala
C2.scala
C4.scala
• inheritance (strong)
• member ref (weak)

20. name hashing
• step 2
A.scala B.scala C1.scala
C3.scala
C2.scala
C4.scala
• collect method signatures
and other information

21. name hashing
• step 3
compile initial invalidation
A.scala B.scala C1.scala
C3.scala
C2.scala
C4.scala

22. name hashing
• step 4
A.scala B.scala C1.scala
C3.scala
C2.scala
C4.scala
inheritance: invalidate dependents

23. name hashing
• step 5
API changes: compare the
signature information before and
after compilation
A.scala B.scala C1.scala
C3.scala
C2.scala
C4.scala

24. name hashing
• step 6
member ref: invalidate only if a
"used name" is affected
A.scala B.scala C1.scala
C3.scala
C2.scala
C4.scala

25. name hashing
package example
class B {
def foo: Unit = {
val a = new A
a.something
}
}
package example
class A {
def something: Int = 0
}

26. name hashing
used names:
example.B -> "A
example.B -> "AnyRef
example.B -> "B
example.B -> "Int
example.B -> "Object
example.B -> "Unit
example.B -> "a
example.B -> "example
example.B -> "example;A;init;
example.B -> "java;lang;Object;init;
example.B -> "scala
example.B -> "something

27. name hashing
package example
class B {
def foo: Unit = {
val a = new A
a.something
}
}
package example
class A {
// was 0
def something: Int = 1
}
• body change doesn't matter

28. name hashing
package example
class B {
def foo: Unit = {
val a = new A
a.something
}
}
package example
class A {
def something: Int = 0
// new
def somethingElse: Int = 1
}
• unused names do not matter

29. name hashing
• implementation body changes
• changes to unused methods / vals
prevents invalidation spread based on

30. compiler bridge

31. compiler bridge
build tool
Zinc
compiler interface (xsbti)
compiler bridge (xsbt)
Scala Compiler (scala.tools.nsc)
compiler bridge source JAR

32. compiler bridge
parser
etc
typer, patmat
etc
Analyzer
API
Extractor
Dependency
refchecks
etc
jvm
pickler

33. compiler bridge
typer, pickler, etc patmat, erasure, etc jvm4.72%
0.00% 25.00% 50.00% 75.00% 100.00%
compiler phases

34. dependency origins

35. dependency origins
subprojects local library JARs

36. dependency origins
subprojects local library JARs
• we have the associated
Analysis to perform
name hashing

37. dependency origins
subprojects local library JARs
• we only need
signature (API)
changes from
upstream

38. next steps

39. liberation from a machine

40. liberation from a machine
2 items
/Users/eed3si9n/work/zinc-example/A.scala -> /Users/eed3si9n/work/zinc-
example/target/scala-2.13/classes/example/A.class
/Users/eed3si9n/work/zinc-example/B.scala -> /Users/eed3si9n/work/zinc-
example/target/scala-2.13/classes/example/B.class
library dependencies:
4 items
/Users/eed3si9n/work/zinc-example/A.scala -> /Users/eed3si9n/.coursier/
cache/v1/https/repo1.maven.org/maven2/org/scala-lang/scala-library/2.13.1/
scala-library-2.13.1.jar
/Users/eed3si9n/work/zinc-example/A.scala -> /Users/eed3si9n/.sdkman/
candidates/java/8.0.222.hs-adpt/jre/lib/rt.jar
/Users/eed3si9n/work/zinc-example/B.scala -> /Users/eed3si9n/.coursier/
cache/v1/https/repo1.maven.org/maven2/org/scala-lang/scala-library/2.13.1/
scala-library-2.13.1.jar
/Users/eed3si9n/work/zinc-example/B.scala -> /Users/eed3si9n/.sdkman/
candidates/java/8.0.222.hs-adpt/jre/lib/rt.jar
library class names:
2 items
/Users/eed3si9n/.coursier/cache/v1/https/repo1.maven.org/maven2/org/scala-
lang/scala-library/2.13.1/scala-library-2.13.1.jar -> scala.Int
/Users/eed3si9n/.sdkman/candidates/java/8.0.222.hs-adpt/jre/lib/rt.jar ->
java.lang.Object

41. liberation from time
2 items
/Users/eed3si9n/work/zinc-example/target/scala-2.13/classes/example/A.class ->
lastModified(1570453146449)
/Users/eed3si9n/work/zinc-example/target/scala-2.13/classes/example/B.class ->
lastModified(1570453146456)

42. repeatable build = build as function

43. compiler integration

44. compiler integration
• incremental compilation as a first-class
feature

45. compiler integration
• incremental compilation as a first-class feature
• test for specific bridge
• no more source downloading
• evolve bridge together with the language

46. compiler integration
parser
etc
typer, patmat
etc
Analyzer
API
Extractor
Dependency
refchecks
etc
jvm
pickler
• API Extractor could probably go away if the bridge
was integrated with the compiler, saving both time
and memory

47. compiler integration
subprojects local library JARs
• incremental compilation with any JAR?

48. compiler integration
subprojects local library JARs
outline types
• we only need
signature (API)
changes from
upstream

49. compiler integration
subprojects local library JARs
outline
typer
typer, pickler etc jvm
refcheck,
erasure
outline
typer
typer, pickler etc jvm
refcheck,
erasure
outline
typer
typer, pickler etc jvm
refcheck,
erasure
outline/foo.jar
outline/bar.jar

50. Thank You