Keynote, ISSRE-13, St. Malo, France, November 4, 2004. Outline: 21st Century IT Trends, Mobile Technology Crisis, Test Effectiveness Levels, Level 4 Case Study, Reliability Arithmetic, Test Performance Envelope.

1. ™
mVerify
A Million Users in a Box ®
Mobile Reliability Challenges
Robert V. Binder
ISSRE 2004
November 2, 2004
www.mVerify.com

2. Overview
 21st Century IT Trends
 Mobile Technology Crisis
 Test Effectiveness Levels
 Level 4 Case Study
 Reliability Arithmetic
 Test Performance Envelope
 Conclusion
© 2004 mVerify Corporation 2

3. The New IT Reality
 Last 25 years  Next 25 years
 Information driven society  User population 100x
 New and better ways to do  New things
old things
 Persistent partial attention
 Focused, skilled interaction
 Mobile, ubiquitous
 Tethered
Very low visibility, very high failure impact
© 2004 mVerify Corporation 3

4. The New IT Reality: Ubiquity
 Cheap fat pipes everywhere
 Optical backbone + wireless MAN, LAN, PAN
 WiMax: 802.16, 802.20
 Ad-hoc Mobile networks
 Cycles and storage
 Moore's law
 Watts per MIPS
 No more shrink-wrap
 Application service provider/subscription model
 Web services, ultra large databases
 Grid computing
 Converging user device form factor
 Cell phone + PDA + pager + Pocket PC + ...
 D-2-B Interfaces – 5 years?
About 10x every five years!
© 2004 mVerify Corporation 4

5. Robert’s Afternoon
Robert’s Afternoon: Seamless Mobility
courtesy Motorola, Inc.
This shows Motorola’s high-level vision.
It is not a product roadmap or indicative of any
specific product/service offering.
© 2004 mVerify Corporation 5

6. The Unchanged IT Reality: Software
 The Next Big Thing
 Extreme Programming ?
 Aspect-oriented languages ?
 Model-driven Architecture ?
 Still no Silver Bullet
 Subtractive component reliability
 Design limited to human ability and organization
 Low-fidelity test suites aren’t effective
Bug barrier: 5/KLOC (pre-test)
any language, any process
© 2004 mVerify Corporation 6

7. Mobile Technology Challenges
 Testing wired apps difficult and expensive
 20% to 50% of all software development $ on testing
 Available test automation technology 10+ years old
 Annual cost to U.S. of inadequate testing: $56 billion
 Testing mobile apps much harder:
 Connectivity: “Can you hear me now?”
 Mobility: location-based services
 Scalability: at least 10x web user population and mobile
 Security: always on, always hackable
 PLUS assure functionality, performance, and integration
Mobile App Fault Space Much Bigger
© 2004 mVerify Corporation 7

8. A True Crisis
Current software technology
CANNOT
achieve reliable mobile apps
© 2004 mVerify Corporation 8

9. What Can Testing Do?
 Test Performance
 Effectiveness (reliability/quality increase)
 Efficiency (average cost per test)
 Levels
 1: Testing by poking around
 2: Manual Testing
 3: Automated Test Script
 4: Model-based
 5: Full Test Automation
Each Level 10x Improvement
© 2004 mVerify Corporation 9

10. Level 1: Testing by Poking Around
Manual
“Exploratory”
Testing
•Low Coverage
•Not Repeatable
•Can’t Scale
•Inconsistent System Under Test
© 2004 mVerify Corporation 10

11. Level 2: Manual Testing
Test Setup
Manual Manual
Test Design/ Test Input
Generation
•1 test per hour
•Not repeatable
Test Results System Under Test
Evaluation
© 2004 mVerify Corporation 11

12. Level 3: Automated Test Script
Test Setup
Manual Test Script
Test Design/ Programming
Generation
•10+ tests per hour
•Repeatable
•High change cost Test Results System Under Test
Evaluation
© 2004 mVerify Corporation 12

13. Level 4: Automated Model-based
Test Setup
Model-based
Test Design/ Automatic
Generation Test
Execution
•1000+ tests per hour
•High fidelity
•Case study
Test Results System Under Test
Evaluation
© 2004 mVerify Corporation 13

14. Level 4 Case Study
 Leading financial market
 3 million transactions per hour
 15 billion dollars per day
 650 KLOC Java, Distributed Services …
 System Test Process & Environment
 Automated, Model-based
 Executable operational profile
 Simulator generates realistic unique test suites
 3 years, version 1.0 live Q4 2001
1,000 to 750,000 unique tests per day
© 2004 mVerify Corporation 14

15. Model-based Testing
Extended Use Case
Mode Machine
Invariant Boundaries
Stealth Requirements Engineering
© 2004 mVerify Corporation 15

16. Simulator
 Discrete event simulation
 Prolog implementation (50 KLOC)
 Rule inversion
 Load Profile
 Time domain variation
 Orthogonal to operational profile
 Each event assigned a "port" and submit time
© 2004 mVerify Corporation 16

17. Automated Run Evaluation
 Oracle accepts output of simulator
 About 500 unique rules
 Verification
 Splainer – result/rule backtracking tool
 Rule/Run coverage analyzer
 Comparator
 Extract transaction log
 Post run database state
 End-to-end invariant
© 2004 mVerify Corporation 17

18. Results
 Revealed about 1,500 bugs over two years
 5% showstoppers
 Five person team, huge productivity increase
 Achieved proven high reliability
 Last pre-release test run: 500,000 events in two
hours, no failures detected
 No production failures
© 2004 mVerify Corporation 18

19. Level 5: Total Automation
Automated
Test Setup
Model-based Automatic
Test Design/ Test
Generation Execution
•10,000 TPH
•Oracle Problem Automated
System Under Test
Test Results
Evaluation
© 2004 mVerify Corporation 19

20. AMATE: Level 5 for Mobile Apps
 Advanced Mobile Application Test Environment
 NIST/ATP funded R&D
 Highly realistic end-to-end mobile testing
 Generate and control
 Signal variation related to mobility
 User behavior related to mobility
 Traffic related to mobility
 Model-based
 Trial Use Q2 2005
© 2004 mVerify Corporation 20

21. Reliability Arithmetic
 Reliability: probability of non-failure
 MTTR: mean time to
 recover, repair, restart …
 Availability: percent up-time
 Availability = 1 / 1 + (MTTR  Reliability)
 99.999% availability = 5 min downtime per year
 “Five nines”
© 2004 mVerify Corporation 21

22. Some Reliability Data Points
Reliability Availability,
(Failures/million hours) 6 min MTTR
NT 4.0 Desktop 82,000 0.999000000
Windows 2K Server 36,013 0.999640000
Common Light Bulb 1,000 0.999990000
Stepstone OO Framework 5 0.999999500
Telelabs Digital Cross Connect 3 0.999999842
© 2004 mVerify Corporation 22

23. Test Automation Envelope
Reliability (Effectiveness)
5 Nines
L4 Level
4 Nines Case 5
Study AMATE
3 Nines
L3
2 Nines
Scripting
L2
1 Nine
Manual
1 10 100 1,000 10,000
Productivity: Tests/Hour (Efficiency)
© 2004 mVerify Corporation 23

24. The Mobile Reliability Challenge
 Mobile App Fault Space Bigger
 Connectivity: “Can you hear me now?”
 Mobility: location-based services
 Scalability: large and mobile
 Security: always on, always hackable
 PLUS assure functionality, performance, and integration
 What can be done?
 Test performance envelope the same
 Test budget the same
© 2004 mVerify Corporation 24

25. Scenario: Manual Testing
 Level 2
manual 5 9s
testing 4 9s
 Mobile app 3 9s
fault space
2 9s
10x bigger
1 9s
 Reliability slips 1 10 100 1,000 10,000
to Level 1
© 2004 mVerify Corporation 26

26. Scenario: Improve Efficiency 10x
 L2 Manual
improves 10x to 5 9s
L3 Scripted
4 9s
 L3 Scripted
3 9s
improves 10x to
L4 Model-based
2 9s
 Expect same
1 9s
average reliability
1 10 100 1,000 10,000
© 2004 mVerify Corporation 27

27. Scenario: Mobile 5 Nines
 Increase L4 5 9s
efficiency 10x
 Realistic mobile
environment 4 9s
 Realistic loading
 Realistic 3 9s
functional profile
 Model-based and
automated 2 9s
 AMATE 1 9s
1 10 100 1,000 10,000
 Expect mobile app
5 Nines
© 2004 mVerify Corporation 28

28. Conclusion
Model-Based mobile testing
CAN
achieve reliable mobile apps
© 2004 mVerify Corporation 29

29. Q&A
© 2004 mVerify Corporation 30