Slides of EclipseCon presentation: Test Confessions - What Eclipsers think and do about testing

1. Test Confessions: What Eclipsers
think and do about Testing
Michaela Greiler and Arie van Deursen
Delft University of Technology

2. A Study to identify…
…what Eclipsers think and do about …
Testing

3. When it comes to testing…
When and why do you adopt a given
testing approach?
How are you going How do you test
to test your GUI? different versions?
How do you test the When do you
integration of need integration
multiple plug-ins? tests?

4. Today’s Objectives
• See test practices in Eclipse
• Understand test adoption criteria
• Derive actionable items
• Foster an open discussion on testing
• Learning from each other

5. Study
• Goal: to get a deep understanding of the
testing practices and the testing culture within
the Eclipse community.
• Interviews with 25 experienced developers or
testers representing well-known open and
closed source projects.
• Design followed Grounded Theory.

6. Grounded Theory
Systematic procedure to discover theory from (qualitative) data
S. Adolph, W. Hall, Ph. Kruchten. Using Grounded theory to study the experience of software development.
Empirical Software Engineering, 2011.
B. Glaser and J. Holton. Remodeling grounded theory. Forum Qualitative Res., 2004.

7. Grounded Theory
Analysis of
– More than 200 testing articles specific 4 Eclipse
– Participation & Analysis of @Eclipse events
– Interviews with 25 experienced professionals
– More than 30 hours
– EMF, SOA Platform, xText, Mylyn, Usus, EclEmma,
GMF, CDO, IMP, Jubula…

8. Theory
Category Emerging Theory
Concept
Code
Constant Comparison
Incident
(Interviews, articles…)

9. Requirements
System Test
Analysis
High Level Integration
Design Test
Detail Unit
Design Test
Implementation
Test Scope: collection of components to be verified

11. Tendency: No Separate Test
Teams

12. Focus on Unit Testing

13. Integration
Tests

14. Integration & System Testing

15. “Try to get to a level that you write unit tests,
always, whenever you can.“
“And write your code in such a way that the structure
and the classes can be tested with unit tests.”
“And then, at max. you use one integration or PDE
tests to probe the code.”
“Ultimately, unit test are our best friends, and
everything else is already difficult.” –P14

16. Who does automated unit testing?

17. Who does automated integration testing?

18. Who does automated cross-feature
integration testing?

19. Who does automated testing for different
Eclipse platform versions?

20. Who does automated combinatorial testing
for OS, versions and plug-ins?

21. Testing Plug-in Integration
Cross-Plug-In Integration?
Combinatorial Testing?
Versioning?

22. Minimal Automated
Integration & System Testing
Why?

23. • Technical Prerequisites
• Integration in Daily
Test Methodology Practice
Adoption Model • Application Knowledge
• Ownership

24. Test Adoption Model
explains explains
Used Unused
Test Test
Practices Practices

25. Technical Prerequisites
• Technical Knowledge
Ready available knowledge, existence of
books, blogs, training, events…
• Application Testability
Mocking, models, devices, code coupling,
working with frameworks…
• Tooling & Test Infrastructure
Set-up test environment, tool maturity,
usability…

26. Integration in Daily Practice
• Test Execution Time
Need of human involvement, duration of test
execution, interruption of other tasks…
• Test Design Time
Availability of time, duration of test design,
integration in daily tasks…
• Perception of Importance
Recognizable benefits, attitude…
• Motivation
Clarity of responsibilities, identification with
responsibility…

27. Application Knowledge
• Design Documents
Developer-centered activity, how formal,
detail level…
• End User Requirements
Well-known, clear end user requirements,
where do you get them from, determinable
from one plug-in….

28. Ownership
• Controllability
Eclipse platform code, control over code
evolution, influence on other plug-ins…
• Responsibility
Overall quality, feel responsible, support
scenarios/use cases…

29. Automated
Automated
Integration &
Unit Testing
System Testing
Technical Prerequisites Goal
Technical Knowledge Available
Application Testability Manageable
Tooling & Test
Infrastructure Usable
Integration in Daily Practice
Test Execution Time Short
Test Design Time Available
Importance High
Motivation
High
Application Knowledge
Design Documents Existing
End User
Requirements Known
Ownership
Controllability High
Responsibility High

30. Compensation Strategy

31. Manual
Testing?

32. “These tests that I do are very simple manual tests,
the real tests are coming from the users, that are
doing all kind of different things with *x+.“—P9

33. But, how to involve the users?

34. Prerequisite for User Involvement
Openness
Open Communication Release Management Extensibility
Feedback Manual Testing Automated Testing
Application Knowledge Alpha & Beta Tester Downstream Projects
& Release Train

35. User Developer
Community Community
Plug-In Characteristic
Open Communication
Extensibility
Release Management
Openness Openness
Manual Automated
Integration & System Integration & System
Test Testing Testing Test
Involvement Involvement

36. Implications: Actionable Items
• For individuals
• For teams
• For tool developers
• For organizations
• For the Eclipse community
• For the research community

37. How can the IDE support integration testing?

38. Product
Solution
What should the process of the tool user look like?

39. How to leverage and organize user involvement?

40. Which technology can support user involvement?

41. Discussion:
Building a Testing Community?
• What should the process of the tool user look
like?
• What should the process of the tool user look
like?
• How to leverage and organize user
involvement?
• Which technology can support community
involvement?

42. Your Opinion!
1. Participate @Survey
www.surveymonkey.com/s/EclipseStudy
2. Come and discuss @BOF
“Testing with, at and for Eclipse”- Wednesday, 20:30, Alameda
Make a chance to win a solar car