What operations correlate with defective configuration scripts? We identify three operations: filesystem, infrastructure provisioning, and user accounts. If you want to find defects, find scripts that do these operations. Details: https://akondrahman.github.io/papers/icst2018_tm.pdf

1. Characterizing Defective Configuration
Scripts Used for Continuous
Deployment
Akond Rahman and Laurie Williams
aarahman@ncsu.edu
https://akondrahman.github.io/

2. Outline
• Motivation
• Goal
• Research Questions
• Methodology
• Dataset
• Empirical Findings
• Summary
2

3. Motivation: Continuous Deployment
3
Continuous
Integration
Integrate software
changes regularly
Continuous
Deployment
Integrate software
changes regularly
Automatically
deploy software
changes rapidly

4. Motivation: Continuous Deployment
https://en.facebookbrand.com/assets/f-logo https://www.stitcher.com/podcast/etsy https://brand.netflix.com/en/assets/
Facebook Etsy Netflix
4

5. Motivation: Continuous Deployment Summit
Continuous Deployment
Summit
(2015, 2016, 2017)
Dr. Chris Parnin Dr. Laurie Williams
5

6. Motivation: Infrastructure as Code ( IaC)
Continuous Deployment Summit
(2015, 2016, 2017)
6
Ansible Chef Puppet

7. Motivation: Defects in IaC scripts
• Defects in IaC scripts can have serious consequences
7https://wikitech.wikimedia.org/wiki/Incident_documentation/20170118-Labs
Wikimedia
Commons

8. Goal
To help software practitioners to prioritize validation
and verification efforts for infrastructure as code (IaC)
scripts by identifying the operations that characterize
defective IaC scripts
8

9. Research Questions
• RQ-1: Which are the operations that characterize
defective infrastructure as code (IaC) scripts? How
frequently do the identified operations appear in IaC
scripts?
• RQ-2: How can we build prediction models for defective
infrastructure as code scripts using text features?
9

10. Background
case $::operatingsystem {
‘CentOS’: {
exec { "install iptables":
command => "/bin/yum install iptables -y",
creates => "/sbin/iptables";
}
}
default: {
fail("cannot install on $::operatingsystem")
}
}
https://puppet.com/company/press-room/media-kit 10

11. Methodology
• Dataset Construction
• Empirical Analysis
11

12. Methodology: Dataset Construction
Extract
Commit
IaC Repositories
12
Rater-1
Rater-2
Disagree?
Mark as
defect or
non-defect
Resolver
Mark as
defect or
non-defect
No
Yes

13. Methodology: Empirical Analysis
Dataset
Feature
Extraction
Correlating
Features
Answer to
RQ1
Answer to
RQ2
Bag of words,
TF-IDF
Principal
Component
Analysis
Qualitative
Analysis
Random Forest,
Parameter Tuning, AUC, F1
13

14. Dataset: Sources
Mozilla Openstack Wikimedia
Commons
https://blog.mozilla.org/opendesign/arrival/ https://www.openstack.org/brand/openstack-logo/
https://commons.wikimedia.org/wiki/Category:Logos
14

15. Dataset
Mozilla Openstack Wikimedia
Time Period Aug, 2011- Sep,
2016
Mar, 2011- Sep,
2016
Apr, 2005 – Sep,
2016
IaC Code Size
(LOC)
30,272 122,083 17,439
Defect-related
Commits
558 of 3074,
18.1%
1987 of 7808,
25.4%
298 of 972,
30.6%
Defective IaC
Scripts
259 of 580
(44.6%)
810 of 1383
(58.5%)
161 of 296
(54.4%)
15

16. Answer to RQ1: Which are the operations that
characterize of defective infrastructure as
code (IaC) scripts?
• Filesystem operations
• Infrastructure provisioning
– Build systems
– Data analytics systems
– Database systems
– Web server systems
• Managing user accounts
16

17. Answer to RQ1: Filesystem operations
17

18. Answer to RQ1: Infrastructure provisioning
• Build systems
18

19. Answer to RQ1: Infrastructure provisioning
• Data analytics systems
19

20. Answer to RQ1: Infrastructure provisioning
• Database systems
20

21. Answer to RQ1: Infrastructure provisioning
• Web server systems
21

22. Answer to RQ1: Managing user accounts
22

23. Answer to RQ1: Frequency
21.7
14.5
23.4
6.9
18.9 17.9
2.5 1.1 1.6
0
5
10
15
20
25
Mozilla Openstack Wikimedia
PercentageofIaCScripts
Filesystem Infrastructure provision User account
23

24. Answer to RQ2: How can we build prediction
models for defective infrastructure as code
scripts using text features?
0.76
0.59
0.68
0.75
0.55 0.56
0
0.2
0.4
0.6
0.8
Mozilla Openstack Wikimedia
MedianAUC
Bag-of-words TF-IDF
0.74
0.71
0.73
0.72
0.74
0.7
0.65
0.7
0.75
Mozilla Openstack Wikimedia
MedianF-Measure
Bag-of-words TF-IDF 24

25. Limitations
• Dataset construction
• Text mining
25

26. Implications
• Raise awareness amongst practitioners when doing
certain operations
• Prioritizing verification and validation efforts based on
identified operations
26

27. Summary
Answer to RQ1: Frequency
21.7
14.5
23.4
6.9
18.9 17.9
2.5 1.1 1.6
0
5
10
15
20
25
Mozilla Openstack Wikimedia
PercentageofIaCScripts
Filesystem Infrastructure provision User account
aarahman@ncsu.edu
akondrahman
https://akondrahman.github.io/
Contact
Answer to RQ2: How can we build prediction
models for defective infrastructure as code
scripts using text features?
0.76
0.59
0.68
0.75
0.55 0.56
0
0.2
0.4
0.6
0.8
Mozilla Openstack Wikimedia
MedianAUC
Bag-of-words TF-IDF
0.74
0.71
0.73
0.72
0.74
0.7
0.65
0.7
0.75
Mozilla Openstack Wikimedia
MedianF-Measure
Bag-of-words TF-IDF
27
Motivation: Defects in IaC scripts
• Defects in IaC scripts can have serious consequences
6https://wikitech.wikimedia.org/wiki/Incident_documentation/20170118-Labs
Wikimedia
Commons