2. Lesson -1 Automation Cost Money
You need people to write scripts
You need testbed to run those scripts (lots of
equipments)
You need people to monitor those test bed
runs, and debug running failures.
You need people to maintain those scripts to
reflect product change
Can you justify all those to your upper
management?
3. You might if you
Show some proof in the front.
Can go to your boss and say: I can reduce
major release cycle from 8 month to 5
month if I do this.
Can go to your boss and say: I can reduce
customer found bugs down by 20% if I do
this.
4. Lesson 2: Automated Testing is Dangerous
Once it is automated, chances are, it will not
be manually executed anymore.
No exploratory testing can be done
If your script has problem, and can not catch
bugs (output is pass even it should be fail), it
will be going into darkness for ever
You may lose your chance to catch bugs for
ever and you don’t know.
5. Lesson 3: No Framework, No Automation
If you have one couple of hundred scripts in
hand, you might be fine without a framework.
You need one if you have thousands of scripts
to manage.
How to know which test case passed/failed?
Where to get a decent running report.
How to group test cases easily.
How to debug
You need one if there are many people develop
scripts in parallel.
6. Lesson 4: Use Standard Script Language
VB/TCL/Perl/Python/Ruby
Customized scripts need learning period
Customized script language – Who maintain
it? No community support
Hard to communicate with others –
developers, other test outside the company
7. Lesson 5: Separate writer and runner
Engineer should not develop script and then
execute script. Script execution is a dedicated
job.
Debug takes time
Test bed problem?
Script problem?
Image regression bug?
Script integration takes time.
Script execution should be a 24/7 factory,
should be a machine. Script is just a by-
product, it is the full version report that you
want
8. Lesson 6: Test Bed Independent is very Important
Separate the writer and runner requires the script
should be testbed independent.
Script should be able to run from testbed to testbed
with minimum change to the test framework
configuration, not to script itself.
Hard coded router/switch names, IP addresses,
interface names are not good when switch testbeds
A handover process is needed between the writer
and runner.
Develop a “script checker” tool to check the hard
coded values in the script as an acceptance criteria.
9. Lesson 7:
Manage Your Scripts the Same as Your Bugs
Script need to have states like bugs
(S) – Submitted: Script is submitted to the
regression team
(A) – Assigned: Script is assigned to a regression
engineer
(I) – Integration: regression engineer is putting it on
the regression test bed.
(P) – Production: Script is in the regression testbed
and will be run on regression testing against release
(F) - Feedback needed: Script has errors, more
feedback from writer needed.
10. Lesson 8: Design You Scripts As Data Driven.
Script need to be data driven
Different data means different test cases.
Test_case.tcl {router1 eth0} {router2 eth0}
{router3 eth0 eth1} {router4 eth0 eth1 eth2}
{mode 1} {phase 1} {traffic 1 speed}
Can test all combinations of Mode, Phase and
Traffic with one single script.
Data generation can be automated too.
Can catch more bugs when vary the data
11. Lesson 9: Log is More Important Than
Script
When fail, most of the time we only look at
the logs, not scripts for debugging.
Read log like read a book
More debugging info dumped when fail.
12. Lesson 10:
test case designer and automator separate?
Don’t use automator who doesn’t respect
testing
Don’t use automator who doesn’t
understand testing
C company use same tester to design,
manual execute and automate the test
cases. And so is J company
13. Lesson 11: User Standard Testbed
DUT2 DUT3 DUT4
VLAN1
VLAN2
VLAN3
VLAN4
Four Ethernet on each of the Device Under Test
1 2 3 4 1 2 3 4 1 2 3 4
被测设备 2
DUT1
1 2 3 4
14. Lesson 11: User Standard Testbed
DUT1 DUT2 DUT3 DUT4
Ethernet1 Open Open Shutdown Shutdown
Ethernet 2 Open Shutdown Open Shutdown
Ethernet 3 Shutdown Open Open Shutdown
Ethernet 4 Shutdown Shutdown Shutdown Shutdown
Logical Topology:
DUT2
VLAN1
VLAN2
VLAN3
DUT1
DUT3
15. Lesson 12:
Dynamic script allocation
Different release has different feature set
Each regression run need to pick different
scripts to run
Tricks for dynamic script allocation
Static file for version matching
Database
16. Lesson 13:
Use Database to keep automation result history
Database for full version report
Database for regression bug debugging
17. Lesson 14:
Script maintenance – version per branch
Certify scripts for each new release
Keep them in a separate code branch