22. Research Tool Software Analytics
“Big Brother” in your IDE
measures testing activities
• 416 Softw. Engineers, 68 countries
• 5 month observation period
• Java / Eclipse
• 13.7 man years of development
observed
Immediate feedback
for developers
23. WatchDog Pre-Test Questionnaire
Can you estimate how much time you spend on
engineering test code versus production code?
50% 50%test code production code
30% 70%test code production code
24. WatchDog Pre-Test Questionnaire
Can you estimate how much time you spend on
engineering test code versus production code?
48% - 52%test code production code
25. After measuring ≥ 3 months
25% - 75%test code
engineering
Most participants
overestimate their
test engineering
activities
production code
engineering
26.
27. Not all Java projects do unit testing
Likely, an underestimation…
28. No test engineering
For projects with test code:
53% of developer did not execute, read, or
modify a single test within five months.
29. Do sets of unit test executions take…
< 5 seconds ≥ 5 seconds
34. Test Driven Development
TDD is not widely practiced. Programmers
who claim to do it, neither follow it strictly nor
for all their modifications.
35. • Be critical: are long-standing
beliefs/insights/claims
• Still valid?
• Valid in your context?
• Triangulate
• Developer opinions deviate greatly from
what we objectively measured
• Mixed-method research
• Longitudinal studies
36. • Estimating is difficult
• Estimating something that you don’t
like is more difficult
• Beliefs are to be questioned all the time
37.
38. Differential diagnosis
- Tests are written, but not run
- Tests are written, but run on Command Line
- Tests are written, but run in Continuous Integration
39. Differential diagnosis
- Tests are written, but not run
- (Tests are written, but run on Command Line)
- Tests are written, but run in Continuous Integration
40. • 1,359 Java and Ruby projects
• 2,640,825 builds
42. Builds with at least one failing test
From 1,108 projects which executed tests
43. Testing #1 reason for build to break
From 1,108 projects which executed tests
Test failure
Other failure
44. Tests are decisive for overall build
From 1,108 projects which executed tests
98.3% projects
Test fails Build breaks
45. Integrating in multiple environments
From 1,108 projects which executed tests
60%projects
Different build result for at least one build in
multi-environment setup
46. Differential diagnosis
- Tests are written, but not run
- (Tests are written, but run on Command Line)
- Tests are written, but run in Continuous Integration
- 60% failing tests in the IDE
- <20% failing tests in CI
- Command Line?
47. ANDY’S LIES
1. 50% of time goes to
testing
2. This talk was on
software analytics
52. From an experienced developer…
“Estimated time working on tests: 20%,
actual time: 6%. Cool statistics! ”
Testing analytics helps developers to get a
better understanding about their own
development behavior!
55. WatchDog 2.0
We are looking into how developers debug
Product improvement: In contact with Eclipse + IntelliJ
New features coming to Eclipse based on our
analytics
Educational improvement: We need to teach our
students to debug more efficiently (with the IDE
debugger)
57. Software Analytics Challenges
for the SPLASH community
• What works in languages? What doesn’t?
• How should we design the next version of
languages?
• How should we design the next generation of
tools?
• How should we improve education?
58. Software
engineering
folklore
Perception w.r.t. dev. testing
Measured developer testing
48% - 52%
25% - 75%
Software analytics
help engineers to
better understand
their own behavior
azaidman
WatchDogTeam
Challenges
for SPLASH
59. Pointers to literature
1. Moritz Beller, Georgios Gousios, Andy Zaidman: How (Much) Do
Developers Test? ICSE 2015: 559-562
2. Moritz Beller, Georgios Gousios, Annibale Panichella, Andy
Zaidman: When, how, and why developers (do not) test in their IDEs.
ESEC/SIGSOFT FSE 2015: 179-190
3. Moritz Beller, Georgios Gousios, Andy Zaidman: Oops, my tests broke
the build: An analysis of Travis CI builds with GitHub. PeerJ PrePrints
4: e1984
4. Fred Brooks: The Mythical Man-Month, Addison-Wesley, 1975
5. Laurent Bossavit: The Leprechauns of Software Engineering, 2012-
2014, https://leanpub.com/leprechauns