Slides presented at the 2018 IEEE International Conference on Software Quality, Reliability & Security in Lisbon, July 2018.

1. Performance Analysis Using
Automatic Grouping
July 18th, 2018
Abderrahmane Benbachir
École Polytechnique de Montréal
Laboratoire DORSAL

2. POLYTECHNIQUE MONTREAL – Francisco, Abder and Prof Michel
● Intro
● Related Work
● Methodology
● Use Cases
● Discussion
● Conclusion
Outline

3. POLYTECHNIQUE MONTREAL – Francisco, Abder and Prof Michel
● Continuous Integration
● Regression tests in several builds
● Performance evaluation
Intro

4. POLYTECHNIQUE MONTREAL – Francisco, Abder and Prof Michel
It is the framework proposed to compare traces using the ECCT and
the critical path. It shows the recorded metrics for a trained
user to find the reason of this specific delay.
Doray, F, and M. R. Dagenais, "Diagnosing Performance Variations by Comparing
Multi-Level Execution Traces", IEEE Transactions on Parallel and Distributed Systems,
vol. pp,issue: 99 no. 1, 2016.
Related Work
TraceCompare

5. POLYTECHNIQUE MONTREAL – Francisco de Melo
Related Work
TraceCompare
POLYTECHNIQUE MONTREAL – Francisco, Abder and Prof Michel

6. POLYTECHNIQUE MONTREAL – Francisco de Melo
● Select groups of comparison manually
● Manual comparison task
● Only two groups of comparison
TraceCompare
POLYTECHNIQUE MONTREAL – Francisco, Abder and Prof Michel
Related Work

7. POLYTECHNIQUE MONTREAL – Francisco de Melo
● Comparative visualization tool
● CCTs
● Several Mathematical applications
Differential Flame
Graph
POLYTECHNIQUE MONTREAL – Francisco, Abder and Prof Michel
Related Work

8. POLYTECHNIQUE MONTREAL – Francisco de Melo
Overview
● Instrument the code
● Run the software with tracing enabled
● Create the tree
● Apply the classification methods
● Compare the groups
POLYTECHNIQUE MONTREAL – Francisco, Abder and Prof Michel
Methodology

9. POLYTECHNIQUE MONTREAL – Francisco de MeloPOLYTECHNIQUE MONTREAL – Francisco, Abder and Prof Michel
Methodology - Data Model

10. POLYTECHNIQUE MONTREAL – Francisco de Melo
Calling Context Model
Construction of tree using lttng and walking technique
POLYTECHNIQUE MONTREAL – Francisco, Abder and Prof Michel
Methodology - Data Model

11. POLYTECHNIQUE MONTREAL – Francisco de Melo
● Instrument the source code
● This will enable recording profiling data which will then be saved
together with each function
● Trace with LTTng and Perf Tools
Metrics = instructions, page faults, cache misses
POLYTECHNIQUE MONTREAL – Francisco, Abder and Prof Michel
Methodology -
Instrumentation

12. POLYTECHNIQUE MONTREAL – Francisco de Melo
Classification
● Classifying a set of metrics by using a non-supervised method
(e.g. k-means and elbow method)
● Analysis of the classification
● Use the result to deduce the root cause
POLYTECHNIQUE MONTREAL – Francisco, Abder and Prof Michel
Methodology - Analyzing the data

13. POLYTECHNIQUE MONTREAL – Francisco de Melo
Methodology - Techniques
POLYTECHNIQUE MONTREAL – Francisco, Abder and Prof Michel
Elbow heuristic
Find the best k
Compare SSE (sum of squared errors)
Comparison: find the biggest gap → starting from 2

14. POLYTECHNIQUE MONTREAL – Francisco de Melo
Methodology - Grouping
POLYTECHNIQUE MONTREAL – Francisco, Abder and Prof Michel

15. POLYTECHNIQUE MONTREAL – Francisco de Melo
Use Case
Cache Optimization
Php Requests
After several executions (100 times), the performance decreased
Periodic problem
POLYTECHNIQUE MONTREAL – Francisco, Abder and Prof Michel

16. POLYTECHNIQUE MONTREAL – Francisco de Melo
Use Case
Cache Optimization
Grouping results relating the caching with the slow executions groups
POLYTECHNIQUE MONTREAL – Francisco, Abder and Prof Michel

17. POLYTECHNIQUE MONTREAL – Francisco de Melo
Use Case
Software Regression (OpenCV)
POLYTECHNIQUE MONTREAL – Francisco, Abder and Prof Michel

18. POLYTECHNIQUE MONTREAL – Francisco de Melo
Use Case
Software Regression (OpenCV)
POLYTECHNIQUE MONTREAL – Francisco, Abder and Prof Michel
Pearson correlation (R) of the variables shows that many metrics are directly
proportional. This mean that the development of linear or multilinear models
using those metrics is not adequate and consequently the comparison of
groups, in a pair-wise approach, can be applied.
Correlation among metrics

19. POLYTECHNIQUE MONTREAL – Francisco de Melo
The differences on each group can indicate normal performance
variations but also can highlight abnormal metric variations
Knowing the abnormal metric and with some knowledge of the
source code, it is possible to find performance regressions
Conclusion
POLYTECHNIQUE MONTREAL – Francisco, Abder and Prof Michel

20. POLYTECHNIQUE MONTREAL – Francisco de Melo
Questions
isnaldo-francisco.de-melo-junior@polymtl.ca
POLYTECHNIQUE MONTREAL – Francisco, Abder and Prof Michel

21. POLYTECHNIQUE MONTREAL – Francisco de Melo
Want to learn more?
POLYTECHNIQUE MONTREAL – Francisco, Abder and Prof Michel

22. POLYTECHNIQUE MONTREAL – Francisco de Melo
Doray, F, and M. R. Dagenais, "Diagnosing Performance Variations by Comparing Multi-
Level Execution Traces", IEEE Transactions on Parallel and Distributed Systems, vol.
pp,issue: 99 no. 1, 2016.
Andrea Adamoli and Matthias Hauswirth. Trevis: A context
tree visualization & analysis framework and its use for classifying
performance failure reports. In SoftVis ’10: Proceedings
of the ACM Symposium on Software Visualization, 2010.
J. M. Spivey. Fast, Accurate Call Graph Profiling. Softw.
Pract. Exper., 34(3):249–264, 2004.
W. N. Sumner, Y. Zheng, D. Weeratunge, and X. Zhang. Precise Calling
Context Encoding. In ACM International Conference on Software Engineering,
2010.
References
POLYTECHNIQUE MONTREAL – Francisco, Abder and Prof Michel