1.
Most slides prepared in collaboration with Ahmed Hassan and Dongmei Zhang
More information available at
https://sites.google.com/site/asergrp/dmse/
Software Mining and Software Datasets
Tao Xie
University of Illinois at Urbana-Champaign
http://taoxie.cs.illinois.edu/
taoxie@illinois.edu

2.
• Associate Professor at University of Illinois at Urbana-
Champaign, USA
• Leads the ASE research group at Illinois
• PC Chair of ISSTA 2015, PC Co-Chair of MSR 2011/2012,
ICSM 2009
• Co-organizer of 2007 Dagstuhl Seminar on Mining
Programs and Processes, 2013 NII Shonan Meeting on
Software Analytics: Principles and Practice
2

3.
Software Services
3

4.
Individual
Social
Isolated
Not much content
generation
Collaborative
Huge amount of artifacts
generated anywhere anytime
4

5.
5
Data pervasive
Long product cycle
Experience & gut-feeling
In-lab testing
Informed decision making
Centralized development
Code centric
Debugging in the large
Distributed development
Continuous release
… …

6.
http://msrconf.org
An international effort to
make software repositories actionable
http://openscience.us/repo/

7.
• Transforms static record-
keeping repositories to active
repositories
• Makes repository data
actionable by uncovering
hidden patterns and trends
12
MailinglistBugzilla Crashes
Field logs CVS/SVN

8.
1313
Field
Logs
Source Control
CVS/SVN
Bugzilla Mailing
lists
Crash
Repos
Historical Repositories Runtime Repos
Code Repos
Sourceforge
GoogleCode

9.
A. Hassan and T Xie. Software Intelligence: Future of
Mining Software Engineering Data. In FoSER 2010.

10.
Software analytics is to enable software practitioners to
perform data exploration and analysis in order to obtain
insightful and actionable information for data-driven
tasks around software and services.
16
Dongmei Zhang, Yingnong Dang, Jian-Guang Lou, Shi Han, Haidong Zhang, and Tao Xie. Software Analytics as
a Learning Case in Practice: Approaches and Experiences. In MALETS 2011
http://research.microsoft.com/en-us/groups/sa/malets11-analytics.pdf

11.
Software analytics is to enable software practitioners to
perform data exploration and analysis in order to obtain
insightful and actionable information for data-driven
tasks around software and services.
17
Dongmei Zhang, Yingnong Dang, Jian-Guang Lou, Shi Han, Haidong Zhang, and Tao Xie. Software Analytics as
a Learning Case in Practice: Approaches and Experiences. In MALETS 2011
http://research.microsoft.com/en-us/groups/sa/malets11-analytics.pdf

12.
18
Research
Topics
Technology Pillars
Target
Audience
Connection to
Practice
Output

13.
19
Software
Users
Software
Development
Process
Software
System
• Covering different areas
of software domain
• Throughout entire
development cycle
• Enabling practitioners to
obtain insights

14.
20
Runtime traces
Program logs
System events
Perf counters
…
Usage log
User surveys
Online forum posts
Blog & Twitter
…
Source code
Bug history
Check-in history
Test cases
…

15.
21
Developer
Tester
Program Manager
Usability engineer
Designer
Support engineer
Management personnel
Operation engineer

16.
• Conveys meaningful and useful understanding
or knowledge towards completing the target
task
• Not easily attainable via directly investigating
raw data without aid of analytics technologies
• Example
– It is easy to count the number of re-opened bugs,
but how to find out the primary reasons for these
re-opened bugs?
22

17.
• Enables software practitioners to come up
with concrete solutions towards completing
the target task
• Examples
– Why bugs were re-opened?
• A list of bug groups each with the same reason of re-
opening
– Which part of my code should be refactored?
• A list of cloned code snippets easily explored from
different perspectives
23

18.
24
Software
Users
Software
Development
Process
Software
System
Information Visualization
Data Analysis Algorithms
Large-scale Computing
Vertical
Horizontal

19.
Bugzilla CVS/SVNMailinglist Crashes
fixed
bug
discussions
Buggy change &
Fixing change
Field
crashes
Estimate fix effort
Mark duplicates
Suggest experts and fix!
New Bug Report

20.
Bugzilla CVS/SVNMailinglist Crashes
fixed
bug
Field
crashes
Suggest APIs
Warn about risky code or bugs
Suggest locations to co-change
New Change
discussions
Buggy change &
Fixing change

21.
Textual Software Artifacts

22.
NL Software Artifacts are of Many Types
• requirements
documents
• code comments
• identifier names
• commit logs
• release notes
• bug reports
• …
• emails discussing bugs,
designs, etc.
• mailing list discussions
• test plans
• project websites & wikis
• …

23.
29
NL Software Artifacts are of Large Quantity
• code comments:
– 2M in Eclipse, 1M in Mozilla, 1M in Linux
• identifier names:
– 1M in Chrome
• commit logs:
– 222K for Linux (05-10), 31K for PostgreSQL
• bug reports:
– 641K in Mozilla, 18K in Linux, 7K in Apache
• …
NL data contains useful information, much of which is not in structured data.

24.
linux/drivers/scsi/in2000.c:
static int in2000_bus_reset(…){ …
reset_hardware(…);
…
}
Code comments contain Specifications
No lock acquisition ⇒ A bug!
linux/drivers/scsi/in2000.c:
/* Caller must hold instance
lock! */
static int reset_hardware(…)
{…}
Tan et al. “/*iComment: Bugs or Bad Comments?*/”, SOSP’07

25.
API documentation contains resource usages
• java.sql.ResultSet.deleteRow() : “Deletes
the current row from this ResultSet object
and from the underlying database”
• java.sql.ResultSet.close() : “Releases this
ResultSet object’s database and JDBC
resources immediately instead of waiting for
this to happen when it is automatically closed”.
java.sql.ResultSet.deleteRow() 
java.sql.ResultSet.close()
Zhong, Zhang, Xie, Mei. Inferring Resource Specifications from Natural Language API Documentation. ASE 2009.

26.
NL Data Contains Useful Info – Example 3
Don’t ignore the semantics of identifiers
Sridhara, Pollock, Vijay-Shanker. Automatically Detecting and Describing High Level Actions within Methods. ICSE 2011

27.
• Unstructured
– Hard to parse, sometimes wrong grammar
• Ambiguous: often has no defined or precise
semantics (as opposed to source code)
– Hard to understand
• Many ways to represent similar concepts
– Hard to extract information from
/* We need to acquire the write IRQ lock before calling ep_unlink(). */
/* Lock must be acquired on entry to this function. */
/* Caller must hold instance lock! */

28.
• Redundant data
• Easy to get “good” results for simple tasks
– Simple algorithms without much tuning effort
• Evolution/version history readily available
• Many techniques to borrow from text
analytics: NLP, Machine Learning (ML),
Information Retrieval (IR), etc.

29.
Stepping Back ….
Image from https://www.snowfactor.com/events/back-to-the-future-quiz/

30.
https://www.kaggle.com/c/the-allen-ai-science-challenge
“consistently understand and correctly
answer general questions about the world.”
“Using a dataset of multiple choice question and answers from a
standardized 8th grade science exam, AI2 is challenging you
to create a model that gets to the head of the class.”

31.
A. E. Hassan and T. Xie:
Mining Software Engineering
Data
• Bug report image
• Overlay the triage questions
Duplicate?
Bugzilla: open source bug tracking tool
http://www.bugzilla.org/
http://www.cs.ubc.ca/labs/spl/projects/bugTriage.html
Assigned To: ?
Anvik, Hiew, Murphy. Who should fix this bug? ICSE 2006.
Wang, Zhang, Xie, Anvik, Sun. An Approach to Detecting Duplicate Bug Reports using Natural Language and Execution Information. ICSE 2008.

32.
Thummalapenta, Sinha, Singhania, Chandra. Automating Test Automation Suresh. ICSE 2012.

33.
Thummalapenta, Sinha, Singhania, Chandra. Automating Test Automation Suresh. ICSE 2012.

34.
Selected units
Selection
threshold
Example of selection based approach from MS Word

35.
conversational structure
…

36.
…Rastkar, Murphy, Murray. Summarizing software artifacts: A case study of bug reports. ICSE’ 10.

37.
Security bug report
“An attacker can exploit a buffer overflow by
sending excessive data into an input field.”
Mislabeled security bug report
“The system crashes when receiving excessive
text in the input field”
Two bug reports describing a buffer overflow
M. Gegick, P. Rotella, T. Xie. Identifying Security Bug Reports via Text Mining: An Industrial Case Study. MSR’10

38.
Term Bug
Report 1
Bug
Report 2
Bug
Report 3
Attack 1 0 1
Buffer
Overflow
1 0 0
Vulnerability 3 0 0
…
Term-by-document frequency matrix quantifies a document
M. Gegick, P. Rotella, T. Xie. Identifying Security Bug Reports via Text Mining: An Industrial Case Study. MSR’10
Start List
Label: Security Label: Non-Security Label:?

39.
A HCP should not change patient’s
account.
An [subject: HCP] should not [action:
change] [resource: patient’s account].
ACP Rule
EffectSubject Action Resource
HCP UPDATE -
change
patient’s
account
deny
Linguistic Analysis
Model-Instance Construction
Transformation
Xiao, Paradkar, Thummalapenta, Xie. Automated Extraction of Security Policies from Natural-Language Software Documents. FSE 2012.

40.
Example Repository:
Project Communication – Mailing lists

41.
A. E. Hassan and T. Xie:
Mining Software Engineering
Data
54
• Most open source projects communicate
through mailing lists or IRC/IM channels
• Rich source of information about the inner
workings of large projects
• Discussions cover topics such as future plans,
design decisions, project policies, code or
patch reviews
• Social network analysis could be performed on
discussion threads

42.
A. E. Hassan and T. Xie:
Mining Software Engineering
Data
55
• Study the content of messages before and after a release
• Use dimensions from a psychometric text analysis tool:
– After Apache 1.3 release there was a drop in optimism
– After Apache 2.0 release there was an increase in sociability
Rigby, Hassan. What can OSS mailing lists tell us? a preliminary psychometric text
analysis of the apache developer mailing list. MSR 2007.

43.
A. E. Hassan and T. Xie:
Mining Software Engineering
Data
56
• When will a developer be invited to join a
project?
– Expertise vs. interest
Bird, Gourley, Devanbu, Swaminathan, Hsu. Open borders? immigration in open source projects. MSR 2007.

44.
Example Repositories:
Source Control and Bug Repositories

45.
A. E. Hassan and T. Xie: Mining
Software Engineering Data
58
Source Control Repositories
• A source control system
tracks changes to
ChangeUnits
• Example of ChangeUnits:
– File (most common)
– Function
– Dependency (e.g., Call)
• Each ChangeUnit:
– Records the developer, change
time, change message, co-
changing Units
ChangeListDeveloper
Time
ChangeChangeUnit
Modify
Add
Remove
Change
Type
* .. *
ChangeList
Message
FI
FR
GM
ChangeList
Type
FI: Feature Introduction
FR: Fault Repairing
GM: General Maint

46.
A. E. Hassan and T. Xie:
Mining Software Engineering
Data
59
Determine
Initial Entity
To Change
Change
Entity
Determine
Other Entities
To Change
Consult
Guru for
Advice
New Req., Bug Fix
“How does a change in one source code
entity propagate to other entities?”
No More
Changes
For Each Entity
Suggested Entity

47.
A. E. Hassan and T. Xie:
Mining Software Engineering
Data
60
• Mine association rules from change history
• Use rules to help propagate changes:
– Recall as high as 44%
– Precision around 30%
• High precision and recall reached in < 1mth
• Prediction accuracy improves prior to a
release (i.e., during maintenance phase)
• Better predictor than static dependencies
alone
Zimmermann, Zeller, Weissgerber, Diehl. Mining Version Histories to Guide Software Changes. TSE 2005.
Hassan, Holt. Predicting change propagation in software systems. ICSM2004.

48.
A. E. Hassan and T. Xie:
Mining Software Engineering
Data
61
import org.eclipse.jdt.internal.compiler.lookup.*;
import org.eclipse.jdt.internal.compiler.*;
import org.eclipse.jdt.internal.compiler.ast.*;
import org.eclipse.jdt.internal.compiler.util.*;
...
import org.eclipse.pde.core.*;
import org.eclipse.jface.wizard.*;
import org.eclipse.ui.*;
14% of all files that import ui packages,
had to be fixed later on.
71% of files that import compiler packages,
had to be fixed later on.
Schröter, Zimmermann, Zeller. Predicting Component Failures at Design Time. ISESE 2006.

49.
A. E. Hassan and T. Xie:
Mining Software Engineering
Data
62
• Given a change can we warn a developer that
there is a bug in it?
– Recall/Precision in 50-60% range
Kim, Zimmermann, Pan, Whitehead. Automatic identification of bug-introducing changes. ASE 2006.

50.
A. E. Hassan and T. Xie:
Mining Software Engineering
Data
63
Percentage of bug-introducing changes for Eclipse
Don’t program on Fridays ;-)
Sliwerski, Zimmermann, Zeller. Don’t Program on Fridays! How to Locate Fix-Inducing Changes. WSR 2005.

51.
A. E. Hassan and T. Xie:
Mining Software Engineering
Data
64
Failure is a 4-letter Word
Zeller, Zimmermann, Bird. Failure is a four-letter word: a parody in empirical research. PROMISE 2011.

52.
A. E. Hassan and T. Xie:
Mining Software Engineering
Data
65
Failure is a 4-letter Word
Zeller, Zimmermann, Bird. Failure is a four-letter word: a parody in empirical research. PROMISE 2011.

53.
• “Cross-validation is inappropriate for estimating
the performance of change classification because
the data points, i.e., changes, follow a certain
order in time.”
• “Randomly partitioning the data set may cause a
model to use future knowledge which should not
be known at the time of prediction to predict
changes in the past.”
– E.g., use information on a change committed in 2014
to predict whether a change committed in 2012 is
buggy or clean.
Tan, Tan, Dara, Mayuex. Online defect prediction for imbalanced data. ICSE 2015 SEIP.
Ye, Bunescu, Liu. Learning to rank relevant files for bug reports using domain knowledge. FSE 2014.

54.
J. Czerwonka, R. Das, N. Nagappan, A. Tarvo, A. Teterev. CRANE: Failure Prediction, Change
Analysis and Test Prioritization in Practice - Experiences from Windows. ICST 2011.

55.
Example Repositories:
Program Source Code

56.
A. E. Hassan and T. Xie:
Mining Software Engineering
Data
69
Source data Mined info
Variable names and function names Software categories
[Kawaguchi et al. 04]
Statement seq in a basic block Copy-paste code
[Li et al. 04]
Set of functions, variables, and data
types within a C function
Programming rules
[Li&Zhou 05]
Sequence of methods within a Java
method
API usages
[Xie&Pei 06]
API method signatures API Jungloids
[Mandelin et al. 05]

57.
• Tons of papers published in the past decade
• Many years of International Workshop on
Software Clones (IWSC) since 2006
• Dagstuhl Seminars
– Software Clone Management towards Industrial
Application (2012)
– Duplication, Redundancy, and Similarity in Software
(2006)
70
Source: http://www.dagstuhl.de/12071

58.
• Motivation
– Copy-and-paste is a common developer behavior
– A real tool widely adopted internally and externally
• XIAO enables code clone analysis in the following
way
– High tunability
– High scalability
– High compatibility
– High explorability
71Dang, Zhang, Ge, Chu, Qiu, Xie. XIAO: Tuning Code Clones at Hands of Engineers in Practice. ACSAC 2012.

59.
72
• Intuitive similarity metric
• Effective control of the degree of syntactical differences between two
code snippets
• Tunable at fine granularity
• Statement similarity
• % of inserted/deleted/modified statements
• Balance between code structure and disordered statements
for (i = 0; i < n; i ++) {
a ++;
b ++;
c = foo(a, b);
d = bar(a, b, c);
e = a + c; }
for (i = 0; i < n; i ++) {
c = foo(a, b);
a ++;
b ++;
d = bar(a, b, c);
e = a + d;
e ++; }
Dang, Zhang, Ge, Chu, Qiu, Xie. XIAO: Tuning Code Clones at Hands of Engineers in Practice. ACSAC 2012.

60.
73
1. Clone navigation based on source tree hierarchy
2. Pivoting of folder level statistics
3. Folder level statistics
4. Clone function list in selected folder
5. Clone function filters
6. Sorting by bug or refactoring potential
7. Tagging
1 2 3 4 5 6
7
1. Block correspondence
2. Block types
3. Block navigation
4. Copying
5. Bug filing
6. Tagging
1
2
3
4
1
6
5
Dang, Zhang, Ge, Chu, Qiu, Xie. XIAO: Tuning Code Clones at Hands of Engineers in Practice. ACSAC 2012.

61.
74
Quality gates at milestones
• Architecture refactoring
• Code clone clean up
• Bug fixing
Post-release maintenance
• Security bug investigation
• Bug investigation for sustained
engineering
Development and testing
• Checking for similar issues before
check-in
• Reference info for code review
• Supporting tool for bug triage
Online code
clone search
Offline code
clone analysis
Dang, Zhang, Ge, Chu, Qiu, Xie. XIAO: Tuning Code Clones at Hands of Engineers in Practice. ACSAC 2012.

62.
75
Available in Visual Studio 2012
Searching similar snippets
for fixing bug once
Finding refactoring
opportunity
Dang, Zhang, Ge, Chu, Qiu, Xie. XIAO: Tuning Code Clones at Hands of Engineers in Practice. ACSAC 2012.

63.
76
Code Clone Search service integrated into
workflow of Microsoft Security Response Center
Over hundreds of million lines of code indexed
across multiple products
Real security issues proactively identified and
addressed
Dang, Zhang, Ge, Chu, Qiu, Xie. XIAO: Tuning Code Clones at Hands of Engineers in Practice. ACSAC 2012.

64.
Combined Security Update for Microsoft Office, Windows, .NET Framework,
and Silverlight, published: Tuesday, May 08, 2012
3 publicly disclosed vulnerabilities and seven privately reported involved.
Specifically, one is exploited by the Duqu malware to execute arbitrary code
when a user opened a malicious Office document
Insufficient bounds check within the font parsing subsystem of win32k.sys
Cloned copy in gdiplus.dll, ogl.dll (office), Silver Light, Windows Journal viewer
Microsoft Technet Blog about this bulletin
“However, we wanted to be sure to address the vulnerable code wherever it
appeared across the Microsoft code base. To that end, we have been working
with Microsoft Research to develop a “Cloned Code Detection” system that we
can run for every MSRC case to find any instance of the vulnerable code in any
shipping product. This system is the one that found several of the copies of
CVE-2011-3402 that we are now addressing with MS12-034.”
77Dang, Zhang, Ge, Chu, Qiu, Xie. XIAO: Tuning Code Clones at Hands of Engineers in Practice. ACSAC 2012.

65.
Software Data Sets

66.
A. E. Hassan and T. Xie:
Mining Software Engineering
Data
79
• PROMISE repository
– http://openscience.us/repo/
• Boa
– http://boa.cs.iastate.edu/
• FLOSSmole:
– http://flossmole.org/
• Software-artifact infrastructure repository:
– http://sir.unl.edu/portal/index.html
• Socorro: Mozilla Crash Stats project
– https://wiki.mozilla.org/Socorro

67.
A. E. Hassan and T. Xie:
Mining Software Engineering
Data
80
• FLOSSmole
– provides raw data about open source projects
– provides summary reports about open source projects
– integrates donated data from other research teams
– provides tools so you can gather your own data
• Data sources
– Sourceforge
– Freshmeat
– Rubyforge
– ObjectWeb
– Free Software Foundation (FSF)
– SourceKibitzer
http://flossmole.org/

68.
A. E. Hassan and T. Xie:
Mining Software Engineering
Data
81

69.
A. E. Hassan and T. Xie:
Mining Software Engineering
Data
82
Yearly MSR Challenge Since 2006
• http://2016.msrconf.org/#/challenge (Boa data for SourceForge, GitHub)
• http://2015.msrconf.org/challenge.php (Stackoverflow data)
• http://2014.msrconf.org/challenge.php (GitHub data)
• http://2013.msrconf.org/challenge.php (Stackoverflow data)
• http://2012.msrconf.org/challenge.php (Change/bug report data for Android)
• http://2011.msrconf.org/msr-challenge.html (Eclipse, Netbeans, Firefox,
Chrome data)
• http://msr.uwaterloo.ca/msr2010/challenge/ (FreeBSD, GNOME
Debian/Ubuntu)
• http://msr.uwaterloo.ca/msr2009/challenge/ (GNOME)
• http://msr.uwaterloo.ca/msr2008/ (Eclipse)
• http://msr.uwaterloo.ca/msr2007/challenge/ (Eclipse, Firefox)
• http://msr.uwaterloo.ca/challenge/ (PostgreSQL, ArgoUML)
• http://msr.uwaterloo.ca/msr2005/
• http://msr.uwaterloo.ca/msr2004/

70.
A. E. Hassan and T. Xie:
Mining Software Engineering
Data
83
• PROMISE repository
– http://openscience.us/repo/
• TraceLab
– http://www.coest.org/index.php/tracelab/about-tracelab
• Apache SVN commits on Github
– https://github.com/monperrus/apache-commits/
• Benchmarks for software maintenance tasks
– http://www.cs.wm.edu/semeru/data/benchmarks/
• Non-functional requirements wordlists
– http://softwareprocess.es/static/What%27s_in_a_Name.html
• Source code ECOsystem Linked Data (SeCold)
– http://www.secold.org/
• Text Analysis for Software Engineering Wiki
– http://textse.wikispaces.com/Home

71.
Must show value before
data quality improves
Correlation vs. Causation

72.
A. E. Hassan and T. Xie:
Mining Software Engineering
Data
91
• Make sure you manually examine the
repositories. Do not fully automate the process!
Image from http://www.quincyma.gov/Government/OCS/neighborhoodtips.cfm

73.
A. E. Hassan and T. Xie:
Mining Software Engineering
Data
92
• Drop all transactions above a large threshold

74.
A. E. Hassan and T. Xie:
Mining Software Engineering
Data
93
• Few developers are given commit privileges
• Actual developer is usually mentioned in the change
message
• One must study project commit policies before
reaching any conclusions
[German 2006]

75.
94
APP DEVELOPERS
APP USERS
App Functional
Requirements
App Security
Requirements
User
Functional
Requirements
User Security
Requirements
informal: app description, etc. permission list, etc.
App Code
Pandita, Xiao, Yang, Enck, Xie. WHYPER: Towards Automating Risk Assessment of Mobile Applications.
USENIX SEC 2013.
Tutorial Slides: http://www.slideshare.net/taoxiease/text-analytics-for-security

76.
• External collaborators: Ahmed Hassan,
Dongmei Zhang, …
• Students…
• Broad colleagues in this area, …
• Funding supports: