1) The document discusses achieving very high reliability for ubiquitous information technology through full test automation. 2) It outlines the new IT reality of growing usage, mobility, and need for high reliability of "five nines" or 99.999% uptime. 3) The strategy proposed is taking a full end-to-end testing approach through automated test generation and execution to achieve the reliability needed for ubiquitous IT to scale to millions of users.

1. Achieving Very High Reliability for
Ubiquitous Information Technology
Center for Embedded Software Systems
Aalborg, Denmark
November 26, 2001
Robert V. Binder
Mobile Systems Verification Corporation
www.MobileSystemsVerification.com

2. Overview
• Vision of very high IT reliability
• The new IT reality and its challenges
• Strategy for very high reliability
© 2001 Mobile Systems Verification Corporation 2

3. The Vision
• Test engineering = system engineering
• Testing is fully automated, high-fidelity
• Very high reliability no longer exotic
© 2001 Mobile Systems Verification Corporation 3

4. Reliability Arithmetic
• Reliability: probability of non-failure
• MTTR: mean time to
– recover, repair, restart …
• Availability: percent up-time
– Availability = 1 / 1 + (MTTR  Reliability)
– 99.999% availability = 5 min per year
– “Five nines”
© 2001 Mobile Systems Verification Corporation 4

5. Some Reliability Data Points
Reliability,
Failures per Availability,
million hours 6 min MTTR
NT 4.0 Desktop 82000.0000 0.999000000
Windows 2000 Server Family 36012.9647 0.999640000
Horseback riding injury 2860.0000 0.999971401
Electric light bulb 1000.0000 0.999990000
Motorcycle riding injury 143.0000 0.999998570
Stepstone OO Framework 5.0000 0.999999950
Residential fire 4.1100 0.999999959
Telecom switch 3.0000 0.999999970
Auto theft 1.0800 0.999999989
Auto accident fatality 0.6680 0.999999993
Commercial airline fatality 0.5000 0.999999995
© 2001 Mobile Systems Verification Corporation 5

6. The New IT Reality: Reliability Sells!
• 372 Google hits on “five nines” “press
release”
– „AT&T Reports Precedent-Setting Five
Nines Performance On Its Market-Leading
Frame Relay Network‟
– „Five Nines: Windows 2000 Servers Deliver
Near-Perfect Reliability for Businesses‟
• Customer retention critical
• “War of the nines” coming soon
© 2001 Mobile Systems Verification Corporation 6

7. The New IT Reality: Usage
• Last 25 years
– New and better ways to do old things
– 1/6th world using cell phones
– Web -- thin client, fat server
• Next 25 years
– Mobile connectivity, new things
– Persistent partial attention
You ain’t seen nothin’ yet
© 2001 Mobile Systems Verification Corporation 7

8. The New IT Reality: Hardware
• A few indicators
– Moore's law
– Watts per MIPS
– Primary, secondary storage capacity
– Wireless networks: personal, local, metro
– 2.5/3 G cellular build-out, adaptive antenna
– Optical data rates, fiber build-out
– Application-specific instruction processors
© 2001 Mobile Systems Verification Corporation 8

9. The New IT Reality: Convergence
• Telecom, datacom converging
• Optical backbone, wireless subnet
– Broadband to the hand
• Remote services, very large databases
• Application service providers
– Executable streaming
– Thin client, fat pipe
© 2001 Mobile Systems Verification Corporation 9

10. The New IT Reality: Mobility
• Converging user device form factor
– Cell phone, PDA, pager, Pocket PC
• Applications
– Location-based
– Peer-to-peer
– Personal agents
• “Digital” interface receding
– Wearable (neck tie, jewelry … )
– Voice recognition
© 2001 Mobile Systems Verification Corporation 10

11. The New IT Reality: Mobility
• Wireless vs. wired connection
– On (off), Connected (not), In-range (out)
– May reconnect with any base station
– Base stations typically not homogenous in
large networks
– Security, privacy can‟t be achieved
physically
Behavior, load, noise, and data link strongly
related to user itinerary
© 2001 Mobile Systems Verification Corporation 11

12. The Unchanged IT Reality: Software
• Still no silver bullet
– Bug barrier  5 per KLOC (pre-test) any
language, any process
– Subtractive component reliability
– Design limited to human bandwidth
– Low-fidelity test suites aren‟t effective
– Hand crafted test suites can‟t scale
© 2001 Mobile Systems Verification Corporation 12

13. The Engineering Challenge
• The new IT reality
– User population grows 100-1000x
– Most users will be mobile
– Infrastructure uncontrollable, unstable
– Increasing feature complexity
– Increasing visibility and impact
– “Five nines” will be required
Existing software technology can’t achieve
high reliability for ubiquitous IT
© 2001 Mobile Systems Verification Corporation 13

14. The Strategy
• The new IT reality
– IT capability/capacity growing rapidly
– Software capability/capacity static
• End-to-end approach can achieve very
high reliability
– Best practices at component scope
– Full test automation at system scope
© 2001 Mobile Systems Verification Corporation 14

15. Testing Process
• Test process
– Design/generation, setup
– Execution, evaluation
• Levels
– Testing by Poking Around
– Manual Testing
– Automated Test Script
– Automated Generation/Agent
– Full Test Automation
© 2001 Mobile Systems Verification Corporation 15

16. Testing by Poking Around
Test Execution
System Under Test
© 2001 Mobile Systems Verification Corporation 16

17. Manual Testing
Test Setup
Test Design/
Generation
Test Execution
Test Results
System Under Test
Evaluation
© 2001 Mobile Systems Verification Corporation 17

18. Automated Test Script
Test Setup
Test Design/
Generation
Test Execution
Test Results
System Under Test
Evaluation
© 2001 Mobile Systems Verification Corporation 18

19. Automated Generation/Agent
Test Setup
Test Design/
Generation
Test Execution
Test Results
System Under Test
Evaluation
© 2001 Mobile Systems Verification Corporation 19

20. Full Test Automation
Test Setup
Test Design/
Generation
Test Execution
Test Results
System Under Test
Evaluation
© 2001 Mobile Systems Verification Corporation 20

21. Test Effectiveness
Tests/Hour Effectiveness Scope
Testing by Poking Around 100 Very low 100
Manual Testing 1 Low 1,000
Automated Test Script 10 Low, repeatable 10,000
Automated Generation/Agent 10,000 Medium 1,000,000
Full Test Automation 1,000,000 High 100,000,000
Full Automation achieves high reliability
promise of profile-based testing
© 2001 Mobile Systems Verification Corporation 21

22. Full Automation Case Study
• E-commerce/securities market over
private tcp/ip net
• 3 million transactions per hour
• 15 billion dollars per day
• 3 years, version 1.0 live 4Q 2001
© 2001 Mobile Systems Verification Corporation 22

23. Full Automation Case Study
• Rational unified process
• About 90 use-cases, 600 KLOC
• Java (services and GUI), some XML
• Oracle DBMS
• Many legacy interfaces
• CORBA/IDL distributed object model
• HA Sun server farm
© 2001 Mobile Systems Verification Corporation 23

24. Full Automation Case Study
• Full Automated Testing
– Extended Use-case and Mode Machine
– Rule-based simulation generates test load
– Test agents (COTS + custom)
– Automate pass/no pass evaluation
• Estimated reliability at five nines
– 1,000,000 events in four hours
– No critical failures
© 2001 Mobile Systems Verification Corporation 24

25. Lessons Learned
• Automatable test requirements are
better than design requirements
• User-based simulation very effective
test approach
• Partial automated evaluation necessary
Full automation can achieve high reliability
for the new IT reality
© 2001 Mobile Systems Verification Corporation 25

26. Conclusion
• Great challenges and opportunities
• High reliability can be achieved
• Very high reliability need not be exotic
© 2001 Mobile Systems Verification Corporation 26