This presentation was made by Glenn Purves, Treasury Board of Canada Secretariat, at the 14th Annual Meeting of the OECD Senior Budget Officials Performance & Results network held at the OECD Conference Centre, Paris, on 26-27 November 2018

1. Centrally-led Performance and
Impact Assessment
Glenn Purves,
Assistant Secretary, Expenditure Management Sector
1
Presentation to the OECD

2. 2
Horizontal Review of Business Innovation Programs
“The Government will initiate a horizontal review of all federal innovation
and clean technology programs across all departments, as federal
innovation programs are dispersed. Consistent with the principles of
Canada’s new Innovation and Skills Plan, the horizontal review will look to
simplify programming and better align resources to improve the
effectiveness of innovation programs”
- Budget 2017
Ensure that programs meet
the needs of firms through
complementary supports,
addressing gaps and
limiting overlap
Improve program alignment
with the goals of the
Inclusive Innovation Agenda
and the Framework for Clean
Growth and Climate Change
Enable programs to better
achieve objectives through
stronger operating policies
and practices
Ensure programs consider
the particular needs,
challenges, and the dual
economic and
environmental role of
clean tech innovation
ALIGN SIMPLIFY ENHANCE CLEAN TECH

3. Scope of Business Innovation Review
Financial
Crown Corporations
Fundamental
Science Spending
Tax Expenditures
The Business Innovation Review was focused on a sub-set of
Government activities that support business innovation…
3
Regulations
General
Government
Procurement
Framework Policies
(e.g. Intellectual
Property)
Sectoral Trade
Promotion
Support provided
directly & in partnership
with third parties
Funding, advice, &
service (existing and
new)*
Industry facing
government R&D
Tech development through
commercialization & exports
Transfer Payments to Provincial &
Territorial Governments
(e.g., Agriculture Programs)
Community Economic
Development
Provincial &
Territorial Programs
Labour Market & Skills
Foreign Direct
Investment Attraction
Industrial
Technological
Benefits

4. Key Findings: Performance measurement challenges
4
ADMINISTRATIVE DATA IS
UNDERUTILIZED IN MEASURING
IMPACTS ON FIRMS
OF DETAILED
HISTORICAL SPENDING
DATA MATCHED TO
ENTERPRISES IN THE
BUSINESS REGISTER
PROGRAM EVALUATIONS
PROVIDED LIMITED INSIGHT
INTO RELATIVE
PERFORMANCE
OPPORTUNITY
FOR DATA-
DRIVEN
EVALUATION
APPROACHES
DATA
OF FUNDING
DIRECTED TO
THIRD PARTY
SERVICE
PROVIDERS
LIMITED UNDERSTANDING OF
OUTCOMES FOR ONE THIRD OF
FUNDING
PROGRAM OVERLAP
CREATES COMPLEXITY FOR
FIRMS
OF PROGRAMS
HAVE SOME DEGREE OF
MANDATE OVERLAP BY
DESIGN OR PRACTICE
26% 27%
47%
Significant opportunities exist
to create cleaner and more
focused mandates
PERFORMANCE MEASUREMENT
PRACTICES FOUND TO BE
VARIED WITH SIGNIFICANT
OPPORTUNITIES FOR
IMPROVEMENT
STRONG
WEAK
MODERATE

5. 5
Horizontal Review of Business Innovation and Clean Technology
Programs observed* a need for:
… common and comparable assessment of program performance,
… employing a horizontal approach,
… enabling greater coordination of policy research, evaluation
approaches, and performance data,
… to better inform resource allocation to and across programs.
The Review demonstrated potential for:
• Common measurement methods and indicators to enable
comparison and benchmarking across programs
• Statistical analysis of program administrative data to strengthen
measurement of program performance
Challenges in horizontal performance measurement
* These observations were also noted in the Review of Federal Support to Research and Development – Expert Panel Report (Jenkins et al., 2011)

6. 6
BUDGET 2018: Business Innovation Program Reform
Funding and
advisory services to
support research
and development
activities of SMEs
National Research Council
– Industrial Research
Assistance Program
Large scale funding
of strategic
innovation
initiatives and
networks
Innovation, Science, and
Economic Development –
Strategic Innovation Fund
Funding and
advisory services to
build and support
export capacity of
innovative firms
Global Affairs Canada –
Trade Commissioner
Service
Foster economic
growth in
communities and
support technology
adoption
Regional Development
Agencies
Creation of four flagship program platforms
As a result of the review, proposed changes based on three principles:
Business-centric lens
Focus on growing
high-potential firms
Accountability for results
Approximately
92
Program Streams
Up to
Reduction
2/3

7. 7
Placing evidence at the centre of program evaluation and design
The Budget 2018 commitment
“With these ambitious reforms, to ensure business innovation programing is
investing in the right place, supporting high-performing businesses and
meeting economic objectives, the Government proposes to provide:
• $1 million per year ongoing, starting in 2018–19, to Statistics Canada to
improve performance evaluations for innovation-related programs.
• $2 million per year ongoing, starting in 2018–19 to the Treasury Board
Secretariat to establish a central performance evaluation team to undertake
innovation performance evaluations on an ongoing basis, including using
the data developed by Statistics Canada.”
-- 2018 Budget Plan, p107

8. Program
administrative data
Enhance common
approaches to data
specification,
development, and
integration
Statistical and
econometric analysis
Advance analytical
techniques informed
by research and
program expertise
Comparable
performance
and impact
Improve horizontal
measurement and
deliver arm’s length,
evidence-based insight
and advice
8
Central Performance and Impact Assessment Unit
Use of statistical approaches, among others, to provide consistent and
comparable measurement of program performance

9. 9
• Canada’s national statistical
agency has a legislative mandate
to collect and compile data that
can strengthen program
evaluation
• Business innovation program
administrative data:
– Financial records of awards
– Summary client service
records
• Linking datasets by common
identifier enables firm-level
analysis of program activities and
impact
Strategic partnership with Statistics Canada
Business
Register
Corporate
Tax Data
Payroll
Data
Import /
Export
Register
Many
others…
Industry
Surveys
Patents

10. -10
0
10
TORONTO
$113M
MONTREAL
$201M
VANCOUVER
$92M
CALGARY
$40M
OTTAWA-
GATINEAU
$40M
EDMONTON
$35M
QUEBEC
$53M
WINNIPEG
$30M
KITCHENER-
WATERLOO-
CAMBRIDGE
$53M
HAMILTON
$21M
TOP10CMAS($,2013)
BusinessinnovationfundingbyCMA
anddifferencebetweenshareof
fundingandshareofGDP
PERCENTAGEPOINTS
Analysing programs through the firms they support
EXPANDING OVER THE LAST 10 YEARS…
Both spending and the number of recipients have increased While programs interacted with twice as
many exporting firms in 2016 than 2010, the
proportion of enterprises relative to the total
shrank
Lower Greater
EXPORTING FIRMS
10
0%
20%
40%
60%
80%
100%
0%
20%
40%
60%
80%
100%
2007 2008 2009 2010 2011 2012 2013 2014 2015 2016
Spending on Innovation Enterprises supported
(as % of 2016)(as % of 2016)
Distribution of business
support varies in relation to
GDP by region (2013)
REGIONALBREAKDOWN
34% 25%
2010 2016
BusinessSupportactivityrelativetoGDP

11. • Natural experiments
in data to generate
counterfactuals to
improve attribution of
outcomes
• Estimate induced
economic impacts for
key indicators
measurable in
individual firms
Econometrics can inform impact assessment

12. 12
The CPIA complements a performance budgeting approach
Programs are aligned
with policy, pursuing
priorities, and designed
for results
Inform policy, identify
program gaps, and
develop opportunities
for greater impact
Programs are high
performing, achieving
results, and meeting the
needs of users
Mission critical
information and insight
on program
performance and reach
Resources are allocated
to achieve greatest
impact
Identify opportunities
and strategies for the
deployment of new
program funding
Providing arm’s length, evidence-based perspective to performance measurement
and evaluation
Polices, programs are
informed by evidence
Programs delivered as
effectively as possible
Resources focused for
greatest impact

13. 13
CPIAU supports and is enabled by data-focused policies
 Open and transparent
government, e.g.,
“committed to openness,
transparency and
collaboration.”
 Focus on outcomes, e.g.,
“evidence-based
decisions that are
anchored in meaningful
data and indicators.”
 “Improve the
achievement of results
across government.”
 “Enhance the
understanding of the
results government
seeks to achieve, does
achieve, and the
resources used to
achieve them.”
 “To collect, compile,
analyse, abstract, and
publish statistical
information”… that is
needed to:
• “develop and evaluate
public policies and
programs”
• “improve … decision-
making for the benefit
of all Canadians.”
Aligned with
Government of Canada
principles
Aligned with Policy on
Results
Aligned with Statistics
Canada mandate and
objectives
Opportunity to advance collaborative, evidence-based policy discourse through the use of
world-class microdata and research capacity

14. 14
The Policy on Results (2016) provides the foundation for results-
based decision-making and allocation of resources based on
performance:
• Evaluation, supported by sound performance measurement
including financial data, is a key component of the Policy on
Results.
• The Policy is the key building block that will allow the
government to move towards performance (-informed)
budgeting.
• Treasury Board Secretariat is responsible for providing
leadership for the performance measurement and evaluation
functions throughout government.
Policy on Results and evidence-based decisions

15. Questions
15