This presentation was performed at AST 2014 (tech.brookes.ac.uk/AST2014/) the 9th International Workshop on Automation of Software Test (AST’14) on June 1, 2014, in Hyderabad, India. It introduces CrowdTesting and provides some initial thoughts on how Crowd Testing and Software Engineering can be combined to get even better results.

1. Università degli Studi dell’Aquila
Henry Muccini
DISIM, University of L’Aquila
henry.muccini@univaq.it, @muccinihenry, www.henrymuccini.com
@AST 2014, Hyderabad, India – June 01, 2014

2. Researcher at the University of L’Aquila, Italy
Research
2

3. How did I get to Crowd Testing?
3
@AST 2012
@MobileSoft2014

4. Acknowledgment and Disclaimer
This work is conducted with the collaboration of
Fabio Guaiani (University of L’Aquila) and the
CLARITER company (Rome).
This presentation represents my group initial effort
towards the understanding of Software Engineering
needs and opportunities in Crowd Testing.
4

5. Agenda
5
Software
Testing Trends
Crowd
Testing
Software
Engineering for
Crowd Testing

6. PREFACE: SOFTWARE TESTING TRENDS
6
Software Testing
Trends
Crowd
Testing
Software
Engineering for
Crowd Testing

7. Software Testing Evolution
7
Software Testing
No dedicated
testing team Dedicated
(in-house)
testing team
Outsourced
testing
(in-house)
resources resources
in the cloud
Software development

8. Investment in QA
8
[World Quality Report 2013-2014]
[1191 respondents]
«Right first time» apps
> $50
billion/year on
applications
testing and
quality
assurance
[Top six trends to drive market for software
testing in 2012,” TechJournal, citing Pierre Audoin
Consultants, Nov 2011]
18%
23%
28%
0% 5% 10% 15% 20% 25% 30%
1
IT budget allocated to QA
2015 2013 2012

9. 9
Software Testing
No
dedicated
testing team
Dedicated
(in-house)
testing team
Outsourced
testing
[1500 respondents]
0% 20% 40%
60%
51%
41%
Internal/In-house Testing
2013 2012
[1500 respondents]
0%
10%
20%
13%
20%
External/Managed Services
2013 2012
0%
20%
40%
8%
26%
"Centralized" Testing
2013 2012
[World Quality Report 2013-2014]

10. 10
Software Testing
No
dedicated
testing team
Dedicated
(in-house)
testing team
Outsourced
testing
[1500 respondents]
0% 20% 40%
60%
51%
41%
Internal/In-house Testing
2013 2012
[1500 respondents]
0%
10%
20%
13%
20%
Crowd Sourced Testing
(External/Managed Services)
2013 2012
0%
20%
40%
8%
26%
"Centralized" Testing
2013 2012
[World Quality Report 2013-2014]
“IDC and Forrester indicate … an increase in
outsourcing of 15% annually over the next
several years.” [Forrester]

11. Testing Mobile App
11
Taken from [World Quality Report 2013-2014], page 27

12. Testing Mobile App
12
Taken from [World Quality Report 2013-2014], page 27
Must be «right first time»
Even «for free» apps
Must get into the market
«first»

13. Summing up…
CROWD TESTING
QA
needs
Mobile
and Web
Outsourci
ng

14. CROWD TESTING
14
Software Testing
Trends
Crowd
Testing
Software
Engineering for
Crowd Testing

15. (the Crowd) [Oxford Dictionaries]
“The mass or multitude of ordinary
people”

16. Crowdsourcing [CrowdSource]
The word is a combination of the words 'crowd' and
'outsourcing'.
Work
Work
Work
Work
Work
Work
Work
Work
Crowdsourcing is the process of getting work or
funding, usually online, from a crowd of people.

17. Crowd Testing…Glossary … (to be refined)
17
Crowd
Crowd
Sourcing
Outsourcing
Crowd Testing

18. What is Crowd Testing?
“ It is, therefore, a form of outsourcing testing tasks to a mass
of internet users (the crowd).” [Stein_TestExp14]
“If you are testing software that all kinds of strangers are going
to use, then why not use a bunch of strangers to test it.”
(Paul Herzlich)
“ Crowtesting puts software to test under real-world conditions by
real users, allowing companies to gather real insight and feedback and to
identify defects quickly and cheaply” [CSTE2013]
18
HM5
HM6

19. Diapositiva 18
HM5 “It’s called the power of the people. I asked thousands of friends to spend 5 minutes testing rather than requiring five of our employees to spend
thousands of hours testing. It got us quicker results on more devices than we typically consider.” [Mike Sparks article]
Henry Muccini; 30/05/2014
HM6 It consists in asking 1000s of crowd testers to spend 5 minutes testing rather than requiring 5 employees to spend 1000s of hours testing.
Henry Muccini; 30/05/2014

20. Examples
19
Tested by a crowd of
9 million people
2 million valuable
comments and
insights
[Stein_TestExp14]
GAP had to scrap
their logo redesign
after protests on
Facebook and
Twitter
[www.pass.ch]

21. 20
diversity/coverage/realistic
independent and test as a user
payment system
immediate results
Principles

22. 21
Crowd
Testing

23. The «players» and the «process»
22
Crowd Testers Crowd Testing
Company
Customers

24. Who are the crowd testers? [RV_TestExp14]
The testing crowd can be a pool of:
▪ Inter-company testers
▪ Intra/inter-project testers in a company
▪ Strategic partners
▪ Domain knowledge experts
▪ Technology experts
▪ Tool experts
▪ Network of end users
▪ Social network users
23
Certified
Specialist
Recreational

25. Crowd Testers expertise and infrastructure
⇒⇒⇒⇒ Crowd Testing types
24
Crowd Testing
Exploratory
Crowd Testing
Enterprise
Crowd Testing

26. Exploratory Crowd Testing
+ Effective to cover all the many configurations
+ Good for non critical systems
+ Simulating real load and testing
- Outcome not predictable
- Discovering basic failures
To help making it more effective:
Show them the big picture of what they are doing
Formal process
A clear report
25

27. Enterprise
cloud
infrastruct
ure
Enterprise Crowd Testing
26
Large enterprises utilizing their in-house excess
capacity for on-demand crowdsourced testing
• Secured private cloud
infrastructure
• Testing Managers
Organized
• Crowd testers screening
• Project management, tracking,
effort planning
Professionalized
• Standardized Process and
ReportingStandardized

28. Enterprise («Managed») Crowd Testing
«Enterprise crowd testing, although still in its nascent
and emerging stage, promises to be the next wave
in the social, mobility and localization era.» [28+_TestExp14]
27
Crowdtesting
Platform
Test
Scheduling
Testers
Profiles
Knowle
dge
center
Comm. and
collaboration
Test
Managem
ent
Reporting

29. A typical Process
28
Preparation Initiation Execution Evaluation Completion
Identify
customer’s
needs
Fix price
Define report
template
Select
crowdtesters
Prepare the test
environment
Initiate testing
Supervise and
monitor
Validate results
Prepare client
deliverables
Testers payment
Presents results
to client
Crowdtesters’
ratings
[TestExp14]

30. Crowd Testing…Glossary
29
Crowd
Crowd
Sourcing
Outsourcing
Crowd Testing
Exploratory
Crowd T.
Managed
Crowd T.
Beta Testing
Enterprise
Crowd T.

31. 30

32. - 1.5 months of full-time
job for one mobile
developer novice + 0.5
months for an expert
mobile developer (client
side)
- 13.000 products in the
DB
- 400 MB of storage into
the device
- DB update
31

33. - The «annulla» feature (when the button is visualized
during the data download) does not work on IOs
- In case the product image is missing, a default image
needs to be loaded
- The «loading %» screen is not fluid
- On IOs, if I download a product with an already
existing image, I get an error
- In the «product list», when tapping on a product
imagine, while looking at the magnified image, I can
still scroll the page (shall be disabled).
32

34. SOFTWARE ENGINEERING FOR CROWD TESTING
33
Software Testing
Trends
Crowd
Testing
Software
Engineering for
Crowd Testing

35. Benefits
34
Availability Coverage
Cost-
effectiveness
Impartiality
Real
scenarios
Speed Diversity

36. Most of the reports come from CrowdTesting
companies, so there could be a «BIAS»!!
35

37. Still… a number of big companies working
on it [BenchmarkQA]
36
are 55.000
testers better
than 1.600 ?
What types of
testing can be
run with the
crowd?
Big companies!

38. Data from Utest (now, «Applause»)
37
Randy Perry. “Trulia Leverages uTest's "Crowdsourced" Testing Solution to Speed Deployment of
High-Quality Mobile Apps.” Sponsored by uTest. January 2013

39. Goldman Sachs Bets Millions On uTest
[http://www.forbes.com/sites/petercohan/2014/01/22/goldman-sachs-bets-millions-on-utest/]
38

40. Challenges in Crowd Testing
39
Security
breaches
Duplicate defect
management
Measurement
Coverage
critical features
coverage
No control
Lack of
ownership
Reward
mechanism
Communication
and mgmt.
Lack of
standards
Continuous
involvement

41. Opportunities for SE research in crowd
testing: COMMUNITIES
Crowdsourcing in SE
Global Software Engineering
Social Software Engineering
Empirical studies
Software Testing
40

42. 1. Opportunities for SE research
Effectiveness of different testing methods
«When»shall we test with the crowd, when not to?
41
Crowd Testing
Cloud
Testing
Lab
Testing
Which sw. domain
Which type of tests Ongoing
Work

43. 2. Opportunities for SE research
Complementing different testing methods
42
Crowd Testing
Cloud
Testing
Lab
Testing
How «crowd and lab testing» can be used
«together»?
Ongoing
Work
Need: clients’ want to get the best
from the combination of both
Efficiency

44. 2. Complementing different testing
methods (cont.)
43
Systematic
Mobile
Testing
Crowd Testing
Solution to
?

45. 2. Complementing different testing
methods (cont.)
44
Systematic
Structural
Testing
Crowd Testing
Solution to
?
Coverage Issues
Test Selection
Test Execution

46. 2. Complementing different testing
methods (cont.)
45
Model-
based
Mobml Testing
Crowd Testing
Model-
based
Mobile

47. Principle
46
Device
independent
tests (in LAB)
+ device
dependent in
crowd
Test HTML 5
PhoneGap
(in LAB)
+ test Phone
Gap app in
Android, iOS,
Microsoft
Test MobML
mobile
(in LAB)
+ test Phone
Gap app in
Android, iOS,
Microsoft
Impl. 1 Impl. With
MBT
2. Complementing different testing
methods (cont.)

48. 3. Opportunities for SE research
Engineering the CrowdTesting Platform
47
Crowdtesting
Platform
Test
Scheduling
Testers
Profiles
Knowle
dge
center
Comm. and
collaboration
Test
Managem
ent
Reporting

49. 4. Opportunities for SE research
«Crowd» Coverage Criteria
48
Measurement
Coverage
critical features
coverage
No control
System Coverage
Crowd Coverage
…
Metrics

50. 5. Opportunities for SE research
Social Software Engineering
How to keep Crowd
Testers motivated
How to keep them
involved
Appropriate reward
mechanisms
49

51. In summary
50
Crowd Testing is growing exponentially
Crowd Testing may bring big benefits to QA
Still, there a number of challenges to be
managed

52. If you are interested to this topic, please
51
Stop by after the presentation
Contact me at henry.muccini@univaq.it
Tweet @muccinihenry
Skype me at henry.muccini
Call me ☺☺☺☺
H. Muccini “Is Crowd Testing (relevant) for Software Engineers?”
@AST20114

53. References «Testing Experience,
25/2014» issue on Crowd testing
[TestExp14] «Testing Experience, 25/2014» issue on Crowd testing
[RV_TestExp14] Rama Murari & Venkat Ramesh Atigadda, “Panorama
of Crowd Testing”, in Testing Experience, 25/2014
[Stein_TestExp14] Markus Steinhauser “Crowd Testing – An Addition
to Traditional Software Testing”, in Testing Experience, 25/2014
[34+_TestExp14] Dieter Speidel, Mayank Mittal & Mithun Sridharan
“Enterprise Crowd Testing”, in Testing Experience, 25/2014, page 34+
[28+_TestExp14] Saisubramanian Sivasailem & Dilipkarthik
Jayamohan, “The Next Wave in the Social, Mobile and Localization Era
- Enterprise Crowd Testing”, in Testing Experience, 25/2014, page 28+
52

54. References
[BenchmarkQA] http://www.benchmarkqa.com/
[CSTE2013] Crowdsourced Software Testing for
Enterprises, August 2013 (by passbrains.com)
[World Quality Report 2013-2014] By Capgemini,
Sogeti, Hp, 5th edition
Crowdsourced Software Testing for Enterprises, August
2013 (passbrain)
Dagstuhl Seminar 13362, Cloud-based Software
Crowdsourcing, Edited by Michael N. Huhns, Wei Li,
and Wei-Tek Tsai
53