O slideshow foi denunciado.
Utilizamos seu perfil e dados de atividades no LinkedIn para personalizar e exibir anúncios mais relevantes. Altere suas preferências de anúncios quando desejar.

Enabling Visual Analytics with Unity - Exploring Regression Test Results in ASIC Verification

330 visualizações

Publicada em

Invited talk at TOCSYC project meeting, November 17, 2017. Västerås, Sweden.

Publicada em: Ciências
  • Seja o primeiro a comentar

  • Seja a primeira pessoa a gostar disto

Enabling Visual Analytics with Unity - Exploring Regression Test Results in ASIC Verification

  1. 1. Research Institutes of Sweden ENABLING VISUAL ANALYTICS WITH - Exploring Regression Test Results in ASIC Verification Markus Borg, Andreas Brytting, Daniel Hansson November 17, 2017 Swedish Institute of Computer Science Software and Systems Engineering Laboratory
  2. 2. 2 Automation in ASIC verification generates large amounts of test results Visual analytics through a game engine helps focusing design and verification effort
  3. 3. Development engineer, ABB, Malmö, Sweden 2007-2010  Editor and compiler development  Process automation PhD student, Lund University, Sweden 2010-2015  Machine learning for software engineering  Bug report management and traceability Senior researcher, RISE SICS AB, Lund, Sweden 2015-  Software engineering for machine learning  V&V for self-driving cars 3 Markus Borg?
  4. 4.  MSc thesis student Andreas Brytting  ASIC verification tool vendor  Research project 4 Credits
  5. 5. 5 WP3 Test effectiveness Task 2 Test result overload
  6. 6.  ASIC design vs. software engineering  Challenge and solution proposal  Tool demo  Evaluation plans 6 Outline
  7. 7. 7 ASIC development ASIC Verification ASIC Design output from design phase register-transfer level (coding) functional correctness probing, temperatures system level, customer applications
  8. 8.  Abstraction  hardware description languages (VHDL, Verilog, SystemVerilog, … )  Version control systems  Test automation  Coverage testing  Steps toward continous integration 8 Software engineering influences ASIC development (NCSU 2017)
  9. 9.  What to verify?  Functional verification  Timing verification  Performance verification  How to verify?  Simulation-based verification  Emulation-based verification (FPGA)  Formal verification 9 ASIC verification
  10. 10.  Directed testing  Manually craft test cases  High maintenance costs  Good for finite condition spaces  Random testing  Automatically generate test cases  Simple to build generator  No maintenance  Needs infinite compute cycles 10 Test case creation constrained random testing & coverage metrics
  11. 11. 11 Peculiarities of ASIC verification Bugs in code escape to silicon => ”re-spin” 70% effort and critical path Exhaustive testing possible? Huge ”test farms” Tool vendors get paid www.erikjohanssonphoto.com
  12. 12.  CPU designs are increasingly complex  New hardware architectures  multicore  multithreading  …  Requirements on small form factors  Trade-off between high performance and low power consumption  Business demands on short time-to- market 12 Contemporary challenges in ASIC verification Purna Mohanty, Aims Technology
  13. 13.  Test automation and regression testing  As often as possible:  Check out the latest source code  Run test cases  Report results  Blame all committers since last pass 13 State-of-practice solution
  14. 14. Large amounts of test results Test result matrices used to hide information 14 The backside… Wikimedia Commons, Brukar:Bep Creative Commons CC-BY 2.5
  15. 15.  One of the largest chip manufacturers worldwide  Development of a CPU  Source code  61,200 files (~3.8 GB)  Verilog, SystemVerilog, assembler, C, and shell scripts  95 committers  Regression testing  500 test cases  2 h execution time  3-8 executions per day 15 Case under study – An ASIC project Wikimedia Commons, User: Zollo, CC-BY 2.0
  16. 16. We target three use cases: 1. General exploration of the design-under-test (DUT) 2. Localization of error-prone parts of the design 3. Detection of potential coverage holes 16 Visual analytics (Illinois Applied Research Institute)
  17. 17.  Why game engine?  Interaction out-of-the-box  Why Unity?  Fairly simple  Scales well  Very popular  Unity???  Cross-platform game engine and IDE  Drag-and-drop 2D and 3D scenes  Scripting in C# 17
  18. 18. 18
  19. 19.  Each building is a file, height represents #commits  Organized in blocks based on folder structure  Color indicates fraction of failed test executions 19 Prototype visualization – test execution cityscape
  20. 20.  Two senior verification engineers in India  Generally positive  First-person shooter controls not necessarily intuitive 20 Prototype evaluation Flickr: billsoPHOTO CC BY-SA 2.0
  21. 21.  Filtering of commit sets  Search lights  Tooltips  Search for folder name  Export to log files 21 Prototype evolved into a tool
  22. 22. 22 Tool demo
  23. 23.  Explore city layouts adhering to physical mapping, i.e., “floor plan”  Evaluate in focus groups with additional verification engineers  Controlled experiment with students  Task: distribute verification effort  Treatment: Cityscape or matrix  Measurement: Time and coverage 23 Future work Flickr: jamesjoel CC BY-ND 2.0
  24. 24. 24 markus.borg@ri.se @mrksbrg mrksbrg.com Automation in ASIC verification generates large amounts of test results Visual analytics through a game engine helps focusing design and verification effort

×