This short talk presents some observations and opportunities in the space of using data analysis to enable, accomplish, and improve software engineering tasks such as testing.

1. My Flings with
Data Analysis
Venkatesh-­‐Prasad
Ranganath
Kansas
State
University
!
Dagstuhl
Software
Development
Analytics
Seminar
June
25,
2014
The
described
efforts
were
carried
out
at
Microsoft
Research
and
Microsoft.

2. Observations
1. Involve the users early
2. Don’t fear the unconventional
3. Embrace dynamic artifacts
• Logs, core dumps, usage profiles, etc.
4. Have a feedback loop involving the users
5. Domain knowledge is a power pellet
6. Questions and answers first, methods later
7. Relevant features and simple methods work really well
8. Quick trumps slow and utility trumps quick
9. Automate (whenever possible)
10.Presentation matters
11.Models are good, simple explanations are better

3. Win8
USB
3.0
Driver
Stack
XHCI
Driver1
Win7
USB
2.0
Driver
Stack
Driver1
EHCI OHCI UHCI
Driver2
USB
2.0
device
USB
2.0
device
Is USB 3.0/Win8 ≈ USB 2.0/Win7?
Collaborators:
Randy
Aull,
Pankaj
Gupta,
Jane
Lawrence,
Pradip
Vallathol,
&
Eliyas
Yakub
When a USB 2.0 device is plugged into a USB 3.0 port on Win8,
the USB 3.0 stack in Win8 should behave as the USB 2.0 stack
in Win7 (along both software and hardware interfaces).

4. One could ….

5. USB2
Log
Pattern
Miner
USB2
Patterns
USB3
Log
Pattern
Miner
USB3
Patterns
Structural
and
Temporal
Pattern
Diffing
USB2
Patterns
USB3
Patterns
DispatchIrp
forward
alternates
with
IrpCompletion
&&
PreIoCompleteRequest
when
IOCTLType=IRP_MJ_PNP(0x1B),IRP_MN_START_DEVICE(0x00),
irpID=SAME,
and
IrpSubmitDetails.irp.ioStackLocation.control=SAME
IOCTLType=URB_FUNCTION_BULK_OR_INTERRUPT_TRA
NSFER(0x09)
&&
IoCallDriverReturn
&&
IoCallDriverReturn.irql=2
&&
IoCallDriverReturn.status=0xC000000E
Patterns-­‐based Compatibility Testing

6. Results
Device Id Detected Reported False +ve Unique
1 9844 478 465 10
2 2545 15 11 2
3 743 4 1 1
4 1372 2 2 0
5 26118 55 55 0
6 26126 0 0 0
7 2320 0 0 0
8 27804 2 1 1
9 34985 115 98 4
10 51556 59 56 3
11 695 0 0 0
12 1372 0 0 0
13 3315 24 23 1
14 9299 3 0 2

7. Test Suite Reduction
Experience-­‐based Selection
Collaborators:
Naren
Datha,
Robbie
Harris,
Aravind
Namasivayam,
&
Pradip
Vallathol

8. Patterns-­‐based Test Suite Reduction
Cluster-­‐n-­‐Select

9. • 50% reduction in test suite size
• 75-­‐80% bugs uncovered (v/s human-­‐based baseline)
• Fully automated (except for threshold setting)
• Flying without safety net
Results

10. Structural & Temporal Patterns (w/ Data flow)
h=fopen
....
fclose(h)
(h!=0
&&
h=fopen)
....
fclose(h)
21
=
fopen(“passwd.txt”,
“r”)
....
fclose(21)
fopen
....
fclose
21=fopen
....
fclose(21)
21=fopen(,
“r”)
....
fclose(21)

11. Observations
1. Involve the users early
2. Don’t fear the unconventional
3. Embrace dynamic artifacts
• Logs, core dumps, usage profiles, etc.
4. Have a feedback loop involving the users
5. Domain knowledge is a power pellet
6. Questions and answers first, methods later
7. Relevant features and simple methods work really well
8. Quick trumps slow and utility trumps quick
9. Automate (whenever possible)
10.Presentation matters
11.Models are good, simple explanations are better

12. Opportunities
• Combining of dynamic and static artifacts/techniques
• Time to move beyond repositories
• Using data analysis to improve techniques
• Exploring/designing language for
• Query
• Visualization
• Scaling
• Distributed Computing
• GPUs

13. Questions