The document proposes a methodology for generating test stimuli using simulation feedback and evolutionary algorithms to improve coverage closure. It describes using an evolutionary algorithm to generate stimuli that maximize coverage metrics based on simulation feedback. Case studies applying this methodology include generating programs for a DLX processor that achieved high code and toggle coverage, generating functional tests for a Pentium 4 that targeted specific performance counters, and generating stimuli for a prototype mobile phone that uncovered bugs. The methodology was found to be broadly applicable and able to find unexpected solutions, though it requires computational resources that can be traded off against quality.
1. Heuristic Stimuli Generation
For Coverage Closure
Exploiting Simulation Feedback
Giovanni Squillero
Politecnico di Torino - Italy
CAD Group ( )
giovanni.squillero@polito.it
2. GOAL
• To propose a methodology for coverage-
directed stimuli generation based on
simulation feedback
• Such stimuli could be added as new content to
improve existing validation suites
DVClub 2010-01-18 giovanni.squillero@polito.it 2
5. Design Choices
• Being able to tackle real problems
• Develop a versatile and broadly applicable
methodology
– Compatible with different environment
– Compatible with any coverage metric
• Minimize effort to change goal/target
– Exploit common aspects
DVClub 2010-01-18 giovanni.squillero@polito.it 5
6. Feedback-Based Approach
• Simulation-based approach
• Exploits feedback from simulation
• Incremental improvement/refinement of the
solution (trial-and-error)
• Trade-off between computational resources
and confidence
• May exploit heuristics or problem-specific
knowledge
DVClub 2010-01-18 giovanni.squillero@polito.it 6
9. Stimuli
• Sequences of bits
• Sequences of keys
• Full fledged assembly language programs
• VHDL test case
• External world
• …
DVClub 2010-01-18 giovanni.squillero@polito.it 9
11. Stimuli Generator
• Exploit an Evolutionary Algorithm to generate
stimuli to maximize a given function
DVClub 2010-01-18 giovanni.squillero@polito.it 11
12. Evolutionary Algorithms
• Meta-heuristic optimization algorithm based
on the concept of population and exploiting
some principles of natural evolution
DVClub 2010-01-18 giovanni.squillero@polito.it 12
13. Evolutionary Algorithms
• Succession of random and deterministic steps
– A systematic way of throwing dices
– Better than pure random
“The great effect produced by the
accumulation in one direction,
during successive generations,
of differences absolutely inappreciable
by an uneducated eye”
DVClub 2010-01-18 giovanni.squillero@polito.it 13
14. Evolutionary Algorithms
• Population
– Multiple solutions considered in each step
– More resistant than pure hill-climbing
– Different solutions may interbreed
DVClub 2010-01-18 giovanni.squillero@polito.it 14
15. Evolutionary Algorithms
• Why using an evolutionary algorithm?
– Adaptative
– Able to find unexpected solutions
– Better than random
DVClub 2010-01-18 giovanni.squillero@polito.it 15
16. Evolutionary Algorithms
• Problem: Fitness function
– GOAL: Optimize the wheel
– FITNESS: Minimize the number of “bumps”
DVClub 2010-01-18 giovanni.squillero@polito.it 16
18. µGP (MicroGP)
• CAD Group general-purpose evolver
– 3 versions (only 2 released under GPL)
– Project started in 2002
– 11 developers + contractors, students, …
• Current version
– ≈ 300 file, > 40,000 lines in C++
DVClub 2010-01-18 giovanni.squillero@polito.it 18
19. µGP (MicroGP)
Evolutionary Optimization: the µGP
toolkit
E. Sanchez, M. Schillaci, G. Squillero
Springer, 2010
ISBN: 978-0-387-09425-0
DVClub 2010-01-18 giovanni.squillero@polito.it 19
21. System & Feedback
Stimuli
Stimuli System
Generator
Feedback
DVClub 2010-01-18 giovanni.squillero@polito.it 21
22. System
• Strongly problem dependant
• Model via simulation/emulation
– HDL (netlist to high-level)
– HW accelerated (e.g., exploiting FPGA)
– Architectural simulator
– ISA simulator
• Real device
DVClub 2010-01-18 giovanni.squillero@polito.it 22
23. Feedback (examples)
• From simulation
– Code coverage metrics (e.g., instruction coverage)
– HW specific metrics (e.g., toggle coverage)
– High-level information (e.g., FSM coverage)
• From the real system
– Performance counters
– Physical measures (e.g., temperature, time, power
consumption)
DVClub 2010-01-18 giovanni.squillero@polito.it 23
31. Post-silicon Verification
• Generate functional test programs for post-
silicon verification
• Activity performed in collaboration
with the ETM Group, Intel (Phoenix)
• Intel Pentium 4
– 42-55 millions of transistors
– 2GHz clock
– NetBurst architecture
DVClub 2010-01-18 giovanni.squillero@polito.it 31
32. System & Feedback
• Real Pentium 4 (no simulation)
• Performance counters as metric
– Introduced in 1993 in IA32 architecture
– 48 event detectors + 18 programmable counters
– Instruction-tagging (for discriminating non-
speculative performance events)
DVClub 2010-01-18 giovanni.squillero@polito.it 32
33. Experimental Results
• Two sets of experiments:
– Mispredicted branches over the total branches
– Clock cycles in which the trace cache is delivering
µops to the execution unit instead of decoding or
building traces (hard metric!)
• Intrinsic features of the µarchitectural design
• Time (both): 12h/program
Haifa 2008 G. Squillero 33
34. Mobile Phone
• 12m-activity performed in collaboration with
Motorola Research Center in Turin
(“Automation of Current Drain Measures”)
DVClub 2010-01-18 giovanni.squillero@polito.it 34
35. System
• Real prototypical mobile phone
– P2K platform
– GSM/3G networks
– Complex applications
• Radio Communication Analyzer, anechoic
chamber
• Controllable power supply
• Custom hardware
DVClub 2010-01-18 giovanni.squillero@polito.it 35
36. Stimuli
• Sequence of keys
• Power supply
• Network status
DVClub 2010-01-18 giovanni.squillero@polito.it 36
38. Experimental Results
• Major bugs discovered in the prototype
– Test infrastructure
– LCD display
– Fake deep-sleep state
DVClub 2010-01-18 giovanni.squillero@polito.it 38
41. Conclusions
• May exploit other methodologies
– Useful starting point
– Effective completion
• May be coupled with other methodologies
– Rule-based instruction randomizers
– Simulation-based approaches
DVClub 2010-01-18 giovanni.squillero@polito.it 41
