Slides from the Eclipse Con North America 2013 presentation

1. EMF Compare 2
Scaling to Millions
Mikaël Barbero 26 March 2013

2. http://idealistpropaganda.blogspot.fr/2010/04/imaginary-foundations-make-difference.html
EMF Compare 2.0
Released mid-se ptember 2012
Sponsore d by Ericsson
under Polarsys and Modeling Platform
Workin g Group (MPWG)

3. http://www.flickr.com/photos/9781230@N06/2270505202
Goal: Performance
Should re late to number of
differenc es, not to the size
of the model
But also simpler API, better understanding of
differences for end-users...

4. Brand New UI Better integration with Team/Compare

5. http://www.flickr.com/photos/viggum/2417430161/
Demo
Because it worths a thousand words

6. How does it works? in a Nutshell

7. Match

8. Two Strategies

9. Two Strategies
Technical ID
XMI:id, resource URI + URI fragment,
your own...

10. Two Strategies
Technical ID Content based
XMI:id, resource URI + URI fragment,
Heuristics, String distance
your own...

11. http://www.flickr.com/photos/brianjmatis/3209203494/
Differencing

12. Rationalization
Longest Common Subsequence
http://www.youtube.com/watch?v=RUckZMzqUcw

13. Dependencies
http://www.flickr.com/photos/qilin/2262821682/

14. Package 1
Addition of ‘Package 1’ within ‘something’
State machine 1 requires the merge of
Addition of ‘State Machine 1’ within ‘Package 1’
Requirements between differences

15. Equivalences
http://www.bavarianfootballworks.com/2012/9/24/3384502/preview-fc-bayern-vs-vfl-wolfsburg

16. Class 1
Setting ‘Property 1’ to reference ‘Associtation 1’
Property 1
the merge is equivalent to
ends ass.
Association 1 Setting ‘Association 1’ to reference ‘Property 1’
‘ass.’ and ‘ends’ are
opposite references
Equivalence between differences

17. http://www.theology21.com/2010/07/27/are-you-talking-to-me-the-art-of-misunderstanding-and-theological-double-talk/
Conflicts

18. Match
Difference
Summary
Requirement
Equivalence
Conflict

19. How to scale to Millions?
http://www.flickr.com/photos/somemixedstuff/2403249501/

20. A graph of EObjects
A «logical» EMF model

21. A
C
D
A graph of EObjects B
G
In Resources
E
F

22. A
C
D
A graph of EObjects B
G
Cross-resources links
E
F

23. A B C
A graph of Resources D
E
G
F

24. A B C
Ancestor
D
E
G
A B C
Left (local) F
D
E
G A B C
F
D
E
G
F
Right (remote)

25. A B C
Ancestor
D
E
G
A B C
Left (local) F
D
E
G A B C
F
D
E
G
F
Right (remote)

26. A B C
Ancestor
D
E
G
A B C
Left (local) F
D
E
G A B C
F
D
E
G
F
Right (remote)

27. A B C
To be Loaded
G E D To be Unloaded
Already browsed
F
Loading Thread
Unloading Thread
Concurrent Resolve All Multi-Threaded

28. A
A B C
To be Loaded
G E D To be Unloaded
Already browsed
F
Loading Thread
Unloading Thread
Concurrent Resolve All Multi-Threaded

29. A B C
To be Loaded
G E D To be Unloaded
Already browsed
F
Loading Thread
Unloading Thread
Concurrent Resolve All Multi-Threaded

30. A B C
To be Loaded
G E D To be Unloaded
Already browsed
F
Loading Thread
Unloading Thread
Concurrent Resolve All Multi-Threaded

31. A B C
To be Loaded
G E D To be Unloaded
Already browsed
F
Loading Thread
Unloading Thread
Concurrent Resolve All Multi-Threaded

32. B G
A B C
To be Loaded
G E D To be Unloaded
Already browsed
F
Loading Thread
Unloading Thread
Concurrent Resolve All Multi-Threaded

33. B G
A B C
To be Loaded
A
G E D To be Unloaded
A
Already browsed
F
Loading Thread
Unloading Thread
Concurrent Resolve All Multi-Threaded

34. B G
A B C
To be Loaded
A
G E D To be Unloaded
A
Already browsed
F
Loading Thread
Unloading Thread
Concurrent Resolve All Multi-Threaded

35. G
A B C
To be Loaded
A
G E D To be Unloaded
A
Already browsed
F
Loading Thread
Unloading Thread
Concurrent Resolve All Multi-Threaded

36. G
A B C
To be Loaded
A
G E D To be Unloaded
A
Already browsed
F
Loading Thread
Unloading Thread
Concurrent Resolve All Multi-Threaded

37. G
A B C
To be Loaded
A
G E D To be Unloaded
A
Already browsed
F
Loading Thread
Unloading Thread
Concurrent Resolve All Multi-Threaded

38. G C
A B C
To be Loaded
A
G E D To be Unloaded
A
Already browsed
F
Loading Thread
Unloading Thread
Concurrent Resolve All Multi-Threaded

39. G C
A B C
To be Loaded
A B
G E D To be Unloaded
A B
Already browsed
F
Loading Thread
Unloading Thread
Concurrent Resolve All Multi-Threaded

40. G C
A B C
To be Loaded
A B
G E D To be Unloaded
A B
Already browsed
F
Loading Thread
Unloading Thread
Concurrent Resolve All Multi-Threaded

41. G C
A B C
To be Loaded
B
G E D To be Unloaded
A B
Already browsed
F
Loading Thread
Unloading Thread
Concurrent Resolve All Multi-Threaded

42. G C
A B C
To be Loaded
B
G E D To be Unloaded
A B
Already browsed
F
Loading Thread
Unloading Thread
Concurrent Resolve All Multi-Threaded

43. G C
A B C
To be Loaded
B
G E D To be Unloaded
A B
Already browsed
F
Loading Thread
Unloading Thread
Concurrent Resolve All Multi-Threaded

44. G C
A B C
To be Loaded
B
G E D To be Unloaded
A B
Already browsed
F
Loading Thread
Unloading Thread
Concurrent Resolve All Multi-Threaded

45. A B C
To be Loaded
B
G E D To be Unloaded
A B
Already browsed
F
Loading Thread
Unloading Thread
Concurrent Resolve All Multi-Threaded

46. A B C
To be Loaded
B
G E D To be Unloaded
A B
Already browsed
F
Loading Thread
Unloading Thread
Concurrent Resolve All Multi-Threaded

47. A B C
To be Loaded
B
G E D To be Unloaded
A B
Already browsed
F
Loading Thread
Unloading Thread
Concurrent Resolve All Multi-Threaded

48. E D
A B C
To be Loaded
B
G E D To be Unloaded
A B
Already browsed
F
Loading Thread
Unloading Thread
Concurrent Resolve All Multi-Threaded

49. E D
A B C
To be Loaded
B C G
G E D To be Unloaded
A B C G
Already browsed
F
Loading Thread
Unloading Thread
Concurrent Resolve All Multi-Threaded

50. E D
A B C
To be Loaded
B C G
G E D To be Unloaded
A B C G
Already browsed
F
Loading Thread
Unloading Thread
Concurrent Resolve All Multi-Threaded

51. D
A B C
To be Loaded
G
G E D To be Unloaded
A B C G
Already browsed
F
Loading Thread
Unloading Thread
Concurrent Resolve All Multi-Threaded

52. D
A B C
To be Loaded
G
G E D To be Unloaded
A B C G
Already browsed
F
Loading Thread
Unloading Thread
Concurrent Resolve All Multi-Threaded

53. D
A B C
To be Loaded
G
G E D To be Unloaded
A B C G
Already browsed
F
Loading Thread
Unloading Thread
Concurrent Resolve All Multi-Threaded

54. D
A B C
To be Loaded
G
G E D To be Unloaded
A B C G
Already browsed
F
Loading Thread
Unloading Thread
Concurrent Resolve All Multi-Threaded

55. D F
A B C
To be Loaded
G E
G E D To be Unloaded
A B C G E
Already browsed
F
Loading Thread
Unloading Thread
Concurrent Resolve All Multi-Threaded

56. D F
A B C
To be Loaded
G E
G E D To be Unloaded
A B C G E
Already browsed
F
Loading Thread
Unloading Thread
Concurrent Resolve All Multi-Threaded

57. F
A B C
To be Loaded
G E D To be Unloaded
A B C G E
Already browsed
F
Loading Thread
Unloading Thread
Concurrent Resolve All Multi-Threaded

58. F
A B C
To be Loaded
G E D To be Unloaded
A B C G E
Already browsed
F
Loading Thread
Unloading Thread
Concurrent Resolve All Multi-Threaded

59. F
A B C
To be Loaded
G E D To be Unloaded
A B C G E
Already browsed
F
Loading Thread
Unloading Thread
Concurrent Resolve All Multi-Threaded

60. F
A B C
To be Loaded
G E D To be Unloaded
A B C G E
Already browsed
F
Loading Thread
Unloading Thread
Concurrent Resolve All Multi-Threaded

61. F
A B C
To be Loaded
G E D To be Unloaded
A B C G E
Already browsed
F
Loading Thread
Unloading Thread
Concurrent Resolve All Multi-Threaded

62. A B C
To be Loaded
G E D To be Unloaded
A B C G E D
Already browsed
F
Loading Thread
Unloading Thread
Concurrent Resolve All Multi-Threaded

63. A B C
To be Loaded
G E D To be Unloaded
A B C G E D
Already browsed
F
Loading Thread
Unloading Thread
Concurrent Resolve All Multi-Threaded

64. A B C
To be Loaded
G E D To be Unloaded
A B C G E D F
Already browsed
F
Loading Thread
Unloading Thread
Concurrent Resolve All Multi-Threaded

65. A B C
To be Loaded
G E D To be Unloaded
A B C G E D F
Already browsed
F
Loading Thread
Unloading Thread
Concurrent Resolve All Multi-Threaded

66. A B C
To be Loaded
G E D To be Unloaded
A B C G E D F
Already browsed
F
Loading Thread
Unloading Thread
Concurrent Resolve All Multi-Threaded

67. A B C
To be Loaded
F
G E D To be Unloaded
A B C G E D F
Already browsed
F
Loading Thread
Unloading Thread
Concurrent Resolve All Multi-Threaded

68. A B C
To be Loaded
G E D To be Unloaded
A B C G E D F
Already browsed
F
Loading Thread
Unloading Thread
Concurrent Resolve All Multi-Threaded

69. A B C
To be Loaded
G E D To be Unloaded
A B C G E D F
Already browsed
F
Loading Thread
Unloading Thread
Concurrent Resolve All Multi-Threaded

70. Why concurrent loading?
The HDD is the bottleneck, right?

71. Scheduling is crucial
Between loading and unloading threads to
avoid high-memory consumption (our goal)

72. A B C
Ancestor
D
E
G
A B C
Left (local) F
D
E
G A B C
F
D
E
G
F
Right (remote)

73. A B C
Ancestor
D
E
G
A B C
Left (local) F
D
E
G A B C
F
D
E
G
F
Right (remote)

74. A B C
Ancestor
D
E
G
A B C
Left (local) F
D
E
G A B C
F
D
E
G
F
Right (remote)

75. Summary

76. Summary
Build the graph of
resources

77. Summary
Build the graph of
resources
Concurrent
resolve all

78. Summary
Find which
Build the graph of
resources has
resources
changed
Concurrent
resolve all

79. Summary
Find which
Build the graph of
resources has
resources
changed
Concurrent Binary comparison
resolve all of resources

80. Summary
Find which
Build the graph of Compare resource
resources has
resources with differences
changed
Concurrent Binary comparison
resolve all of resources

81. Summary
Find which
Build the graph of Compare resource
resources has
resources with differences
changed
Match, Diff, etc.
Concurrent Binary comparison
without resolving
resolve all of resources
proxies

82. Kill two pigs with one (angry) bird
http://extremehdwallpapers.com/wp-content/uploads/2012/08/angry-birds-wallpaper-hd-.jpg

83. http://www.flickr.com/photos/roberts87/2646193801/
Lightweight How light?

84. http://www.flickr.com/photos/themonnie/2495892146/
Fast How fast?

85. Number of fragments (resources) Number of EObjects Disk Usage (XMI files size)
10000 10000K 1000M
1000 1000K
100M
100 100K
10M
10 10K
Small Medium Large Small Medium Large Small Medium Large
Sample Models The numbers (x2 in 2-ways, x3 in 3-ways)

86. The
promised millions
Number of fragments (resources) Number of EObjects Disk Usage (XMI files size)
10000 10000K 1000M
1000 1000K
100M
100 100K
10M
10 10K
Small Medium Large Small Medium Large Small Medium Large
Sample Models The numbers (x2 in 2-ways, x3 in 3-ways)

87. Small Medium Large
120
90
CPU Usage
Time (sec)
60
Some changes in 1 resource
30
1 10 100
number of changes

88. Small Medium Large
120
90
CPU Usage
Time (sec)
60
1 change in some fragments
30
1 5 10 20
number of changed fragments

89. 600
450
Max Heap (MiB)
Memory Usage
300
Max Heap (-Xmx512m)
150
0
Small Medium Large

90. http://www.flickr.com/photos/exploratorium/4689347618/in/set-72157624060408411 http://www.flickr.com/photos/exploratorium/4680503314/in/set-72157624060408411 http://www.flickr.com/photos/exploratorium/4689346802/in/set-72157624060408411
What’s Next?

91. Model Resolving Scope Narrowing Matching
Differencing Post-Process
100 %
75 %
Time distribution
50 %
25 %
Small 0 %
Nominal
Large

92. Index Cross-Resources Links
http://www.flickr.com/photos/manchesterlibrary/5425247171/

93. Eclipse Papyrus support
http://www.flickr.com/photos/manchesterlibrary/5425247171/

94. Porting Eclipse Papyrus support

95. C
A B C
To be Loaded
G E D To be Unloaded
Already browsed
F
Loading Thread
Unloading Thread
Protect the Integrity of the Model When committing, updating...

96. http://nn.wikipedia.org/wiki/Fil:Postcards_and_magnifying_glass.jpg
Generic Content Based Match Engine Decently Fast

97. http://cdn.memegenerator.net/instances/400x/20451927.jpg
Merge Wizard Help to Resolve Conflict

98. http://www.space.com/17376-sharing-the-light-of-two-suns.html
EMF Compare 2.1 With Kepler (Eclipse 4.3), June 2013

99. What you should remember
New UI
Blazing Fast and
Lightweight
More to come ;)

100. http://www.eclipse.org/emf/compare/
http://www.slideshare.net/mikaelbarbero/