My

1. Thought on Effective Testing
By Qingsong Yao

2. 1. User Scenario driven testing
When writing test plan, separate what with How
– First, only think about the user scenario
– Then, think about how you to test the scenario
When write tests, separate what with how
– First, write use scenario
– Second, implement the test cases which test the
scenario

3. User Scenario Example
• Scenario – Block of upgrade SQL Express during VS
2010 Installation (Priority 0)
• Action:
– When a use install VS 2010 on a AMD64 machine with SSE
2005 install, existing SSE instance can’t upgrade SSE 2008
• Success criteria:
– Make sure that Setup program detect existing SSE instance
correctly.
– Make sure that Setup program disable the upgrade/install
of SSE correctly.
– Make sure SSE is not upgraded if upgrade is not needed.
– Existing SSE instance works after setup.

4. 2. Understand the feature
Knowing:
– how the feature will be implemented.
– the risks in term of implementation.
– the risks in term of the impact on customer.

5. Example of “Risk Factors and
Mitigation strategies”
• We are testing product which is not shipped
by SQL Group.
• Setup is sensitive to change, and easy to be
broken.
• Dependence of SSE, such as MSI 4.5 is one of
the main factors of causing setup failing.
• Huge Test Matrix

6. 3. Define your Test Strategy
• Test Strategy
– How to address the risk areas?
– Manual Test v.s. Automation Test?
• Examples:
– Align Test tasks with Dev tasks.
– For larger dev check-ins, test with a private build
before checking-in.
– Encourage writing programming based tests
instead of writing static Meta tests.

7. Example of Test Strategy
• Manual Testing during Improvement Phase
– Avoid huge investigation of test automation
– We define high-level user scenario Success criteria
and China VAT team help us to verify
• Run test as a real customer to find usability issue.
– Verify UI, setup option, and no. reboots
– Verify localized strings
• For bug fix, we drive user scenario and test
dimension from the nature of the fix
• Long term, will take advance of VS setup
automation in SQL Lab

8. 4. Define test patterns
• Test Types:
– Recorded tests, Scripted tests or Data Driven tests, Declarative
Tests
– Model-based or static tests
• Design Patterns:
– Layer Tests
– Repository Pattern, Factory Patterns, Visitor pattern
• Test Dimensions:
– Pair wise or all combination; Parameterized Tests
• Verification Pattern:
– State Verification; Oracle based Verification; Custom Assertion
• Deal with Dependence:
– Dependence Injection
– Mock Object

9. From User Scenario to Test Case

10. Test Case Example
Test Case: Mismatch between SSE language with VS language
if the language of installed SSE instance does not match
with the localized SSE 2008, we should not provide options
to install SSE.
Test Dimension
– VS Language: ENU
– SSE: SSE 2005, SSE 2008
– SSE Language: JPN
Test Case:
use pict tool to use generate 3 test cases
Verification:
• Verify that user can not choose SSE during VS Setup
• Verify that VS setup succeed without any error

11. Conclusion
1. User Scenario driven testing
2. Understand the Feature to Be Tested
3. Define Test Strategy
4. From User Scenario to Test Case
5. Write good Tests
6. Define measurable success critical

12. Reference: Test Strategy
• http://en.wikipedia.org/wiki/Test_strategy
• http://www.projectperfect.com.au/info_test_
strategy.php
• http://www.satisfice.com/presentations/strat
egy.pdf