Jenkins and vManager can work together to automate verification. Jenkins triggers vManager regressions on code commits and generates verification session input files (VSIFs) dynamically. This provides constant regression and coverage while avoiding manually running tests. The combination gives full visibility into the environment state and fixes issues like compilation breakage much faster than previously.

1. Automating the
Verification
A Synergy Between Jenkins and vManager
Tsvi Mostovicz
Verification Engineer
Board design

2. Automating the verification 2
• About RAD
• The Problems We Faced
• A Lesson from the Software Development Domain - Jenkins
• Regressions and Coverage - Cadence vManager
• Using Jenkins Together with vManager
• Solutions
Agenda

3. Automating the verification 3
• Telecommunications access solutions provider
• Founded in 1981, privately owned
• Global company
• R&D located in Israel, Tel Aviv and Beersheba
• Most of our products are based on FPGAs
About RAD

4. Automating the verification 4
Critical Infrastructure
Our Customers
Mobile | Business | Wholesale Utilities| Transportation| Government
Service Providers

5. Automating the verification 5
• FPGA relies heavily on lab testing: the design is fixed, but the verification is not always
updated
• Regression data is not based on the main source code repository
• Manually running regressions takes a lot of time and effort
• No transparency of the regression data
The Problems We Faced

6. Automating the verification 6
A Lesson from Software Development - Jenkins
• Continuous integration systems have solved this issue in the software realm
• Every change to the source repository triggers compilation, testing and publishing
of binaries
Jenkins
• An open-source tool, originally developed as a side project by Sun under the name Hudson,
it was forked after issues with Oracle over control of the project
Allows triggering of jobs based on a variety of factors
Jobs can be as simple as a simple shell script
Stores reports/logs, can send notifications to any
imaginable consumer

7. Automating the verification 7
• Regression is based on VSIF input file
• Programmatic access via the vAPI
Regressions and Coverage - Cadence vManager
• Manage regressions:
– Run regression
– Analyze metrics
– Track progress
• Client-server model
• Database backend

8. Automating the verification 8
Using Jenkins Together with vManager
• We’re limited by the amount of licenses and hardware resources availability during the day 
only compile the sources on commits to source control
– Protection from non-compiling sources
• Jenkins triggers vManager to run nightly regressions on last known good compilation
– We always have a running regression
– Coverage is continually being aggregated
– Jenkins can generate the VSIF (Verification Session Input File) on the fly for vManager
– Jenkins uses vAPI to “talk” to vManager’s database

9. Automating the verification 9
Source
control
Mail to user
Compile
Env Y
Compile
Env X
Solution – Detailed Flow
// dut.sv
module rx_mac();
...
- wire bla;
+ reg bla;
Compile
Env X
`FEATURE_Z
Regression
Env X
Regression
Env X
`FEATURE_Z
No
testlist.vsif
found
Check for testlist.vsif
@19:00 on
weekdays trigger
appropriate
regressions Generate full vsif and
send regression end
to vManager vManager
Env X session
@7:00 on weekdays
get list of nightly
regressions from
vManager
Regression
report
Env X
User’s debug
environment
Debug test
Mail to user
VSIF
vAPI vAPI
vAPI
Stop list of matching nightly regressions,
cleanup logs and send list to Jenkins
User Jenkins vManager

10. Automating the verification 10
Solution – Creating a VSIF on the Fly
• The description tag can hold HTML tags
– Use any formatting (eg. bold, underline)
– Add hyperlinks to other sources of information
– Very useful for reports
• Using #include allows separation of responsibilities:
– Verification owner maintains simple test list
– Automation takes care of the rest
• Using #define allows different tests to be run depending on design capabilities

11. Automating the verification 11
session eth_ntu.5.2.0.ETX_2I.tm_end2end_nightly {
top_dir : /home/jenkins/regressions;
queuing_policy : round_robin;
output_mode : log_only;
description : <text><p>${DESCRIPTION_HTML}</p>
<p>Defines used in build:<ul>${DEFINES_DESCRIPTION}</ul></p>
<p>Changes since last regression:<ul>${CHANGES}</ul></p></text>;
};
group tests {
pre_run_script : "cp -f $ENV(WORKSPACE)/{*.tcl,*.hex,*.mif,*.dat,project_specific_defines.v}
$DIR(run)/ 2>/dev/null || :";
run_script : "i_run.sh -comp_path=$ENV(WORKSPACE) ${ENV_TYPE} -seed=$ATTR(seed) $ATTR(sim_args)
${ENVIRONMENT} $ATTR(test_name)";
scan_script : "vm_scan.pl ${ENV_FILTER} ius.flt shell.flt";
run_mode : batch;
seed : random;
count : ${COUNT};
timeout : 15000;
#include "testlist.vsif"
};
VSIF Example - Automation

12. Automating the verification 12
VSIF Example – Test List
#ifdef ETX_2I
test tm_end2end_2i_test {};
#elif ETX_2I_B
test tm_end2end_2i_b_test {};
#elif ETX_2I_B_PLUS
test tm_end2end_2i_b_plus_test {};
#endif
test test_mem_map {
count : 1;
};

13. Automating the verification 13
Solution – Using vAPI with Python Requests Library
• Using Python’s requests library requires much less overhead than using Cadence’s Jenkins plugin 
• vAPI access is done by sending a “filter” via a HTTP POST request
• Example: Getting all nightly regressions
nightly_filter = { "filter":
{"@c" : ".ChainedFilter",
"chain" : [
{"operand" : "GREATER_THAN",
"@c" : ".AttValueFilter",
"attValue": int(time_started.strftime("%s")) * 1000,
"attName" : "start_time"},
{"@c" : ".ExpressionFilter",
"exp" : "nightly",
"attName" : "name"} ],
"condition" : "AND"},
"projection":{"selection":["name"]} }
session_list = requests.post(url + 'sessions/list', json=nightly_filter, auth=cred)
Filters are simple
Python dictionaries
Just enter the correct
key/value pair to get
what you want

14. Automating the verification 14
Cadence’s Jenkins Plugin Wishlist
• Really need to fix: usability
– Allow usage of environment variables
– Do NOT force file names upon the user
• Nice to have: help with vAPI
– Dynamic form fill for generating a request
• Wow, you rock:
– Build step: run regression  help me create my VSIF
– Return xUnit formatted results to Jenkins

15. Automating the verification 15
• Jenkins - https://jenkins.io
• vManager – Cadence Help (15.2)
– Incisive vManager
• vAPI – Cadence Help (15.2)
– Incisive Enterprise Simulator – Metric Driven Verification – Incisive vManager – vManager API
• Python request library - http://docs.python-requests.org
Links

16. After
• Instant knowledge on compilation
breakage
• Coverage is constantly being
aggregated
• Full visibility into state of environment
Key Takeaways
Before
• Spend a lot of time getting
environment to compile
• Unused resources of idle machines
• No visibility into the current state of
the environment
• Effort required: 1 engineer X 4 months
• Time until fix of compilation breakage: 3-4 months  2 hours – 2-3 days
• Time to find bug based on a “small” fix: Unknown  1 day

17. Tsvi Mostovicz
Verification Engineer
tsvi_m@rad.com