OECD expert workshop on the measurement of public procurement of innovation. "Using administrative data to measure public procurement of R&D: Opportunities and Challenges", presentation by Lionel Kapff

1. Using administrative data to measure public procurement of R&D: Opportunities and Challenges OECD Expert Workshop Measurement of Public Procurement of Innovation Lionel Kapff Deloitte Consulting lkapff@deloitte.com
Paris, 4th February 2013

2. © 2013 Deloitte
2
Using Administrative Data Sources to Identify Public Procurement of R&D
Contents
Introduction 3 Purpose of the study 4 Methodology 5 Availability of public procurement data across Europe 7 Preliminary interim results for selected European countries 9 Concluding remarks 10 Questions & Answers Annexes:
•Contact information;
•Recommended readings.

3. © 2013 Deloitte
3
Using Administrative Data Sources to Identify Public Procurement of R&D
Introduction
Study commissioned by the European Commission, DG CNECT:
•“Quantifying public procurement of R&D of ICT solutions in Europe” Study consortium:
•t33 + Deloitte Consulting Belgium + Spark Legal Network Measurement of:
•Public procurement of ICT
•Public procurement of R&D
•Public procurement of R&D of ICT solutions Coverage:
•27 EU Member States + Norway + Switzerland
•2008 - 2011 Timeline of the study:
PP
of
R&D
PP
of
ICT
PP of R&D of ICT
1. Methodological analysis
2. Data gathering and analysis
3. Comparative analysis and future methodology
01/2012
04/2012
11/2013
08/2013

4. © 2013 Deloitte
4
Using Administrative Data Sources to Identify Public Procurement of R&D
Purpose of the study
Objectives of the study:
•Establish statistics on the R&D and ICT procurements made across Europe, including those made at the country-level (and hence not covered by TED) for the first time.
•Set up a methodology for updating the statistics, assuring sustainability and re-use of the work already carried out for the first time data gathering exercise. Policy goals:
•Reliable statistics are the basis for evidence-based policy-making.
•Statistics enable policy-makers to:
•Set targets;
•Benchmark the status of their jurisdiction compared to others; and
•Measure progress of implemented policies.
•The European Commission will make use of the measurements of expenditure patterns in order to design and develop PCP / PPI support policies efficiently and effectively over time.

5. © 2013 Deloitte
5
Using Administrative Data Sources to Identify Public Procurement of R&D
Methodology
Funder-based approach:
•Collection of public procurement raw data from administrative data sources (i.e. public procurement notifications and contract award notices) Data gathering from various sources:
•OJEU - TED
•Public procurement authorities at different levels of government (typically: national/regional e-notification platforms for public procurement)
•Private tender alert service providers Computer-based data analysis + manual verification of results Identification of ICT procurements based on CPV codes
•Definition of ICT based on OECD statistical guidelines
•Translation of CPC Rev. 2 list of ICT products into list of relevant CPV codes

6. © 2013 Deloitte
6
Using Administrative Data Sources to Identify Public Procurement of R&D
Methodology
Identification of R&D procurements based on keywords: Missing data (e.g. contract values, defence procurement) is estimated based on:
•Extrapolations of existing data
•Additional statistical input and qualitative information (interviews with Ministries of Defence, public procurement authorities and national statistical offices are carried out in all countries)
1. Manual screening of all CPV-73 procurements in TED for compliance with the OECD Frascati Manual definition of R&D
2. Identification of R&D keywords in the Frascati-compliant TED CPV-73 procurements with a textual analysis software and a manual verification + translation of keywords
3. Automatic screening of public procurement raw data (all CPV codes) for R&D procurements based on R&D keywords in titles and descriptions
4. Manual verification of the results based on the OECD Frascati Manual definition of R&D

7. © 2013 Deloitte
7
Using Administrative Data Sources to Identify Public Procurement of R&D
Availability of public procurement data across Europe Data availability of under EU-threshold public procurement
Sources: Deloitte, t33, Spark Legal Network; interim results of the “Study on e-Procurement measurement and benchmarking” (EC DG MARKT)
Countries with a mandatory one-stop portal
Belgium, Cyprus, Denmark, Estonia, Finland, Hungary, Ireland, Latvia, Lithuania, Luxembourg, Norway, Poland, Portugal, Slovakia, Slovenia
Countries with multiple portals
Bulgaria, Czech Republic, Italy, Malta, Netherlands, Rumania, Spain, Sweden, United Kingdom
Countries with a one-stop portal for national authorities
Austria, France, Switzerland
Countries with a voluntary portal
Germany
Countries with a planned portal
Greece, Netherlands

8. © 2013 Deloitte
8
Using Administrative Data Sources to Identify Public Procurement of R&D
Availability of public procurement data across Europe Data availability and quality issues
Fragmentation of public procurement data
•Public data is not always centrally available.
•There are more than 300 e-notification platforms scattered across Europe. Incomplete coverage
•Several e-notification platforms have only been recently established.
•Some sensible areas – such as defence – are not (fully) covered.
•National thresholds for publication vary across countries. Data quality issues
•Values of contracts are often not indicated.
•CPV codes are often not used for below-EU-threshold procurements. Data gathering issues
•Batch-downloads of full databases or even searches are often not possible.
•Some public and private data providers are reluctant to cooperate.

9. © 2013 Deloitte
9
Using Administrative Data Sources to Identify Public Procurement of R&D
Preliminary interim results for selected European countries R&D public procurement contracts awarded in 2011
•Total of 946 million EUR of R&D procurements identified in 17 countries covered so far.
0-10 million EUR contracting volume
Estonia*, Hungary*, Slovenia, Norway*
21-100 million EUR contracting volume
Austria*, Belgium, Czech Republic*, Romania*, United Kingdom*
11-20 million EUR contracting volume
Finland, Lithuania*, Slovakia*, Switzerland*
>100 million EUR contracting volume
Italy, France*, Poland, Spain*
* Data currently only includes above EU-threshold procurements.

10. © 2013 Deloitte
10
Using Administrative Data Sources to Identify Public Procurement of R&D
Concluding remarks Using administrative data to measure public procurement of R&D
Opportunities
•Administrative data is currently the most detailed, up-to-date and complete data source available on public procurement
•Measures are currently undertaken across Europe to improve the availability, reliability, coverage and comparability of administrative data on public procurement
Challenges
•R&D public contracts other than those where the benefits accrue exclusively to the contracting authority are exempted from publication obligations under the EU Procurement Directives
•Problems with regard to availability, reliability and international comparability of administrative data on public procurement
•Goodhart’s law: Challenges for target-driven measurement

11. © 2012 Deloitte
11
Questions and Answers
Using Administrative Data Sources to Identify Public Procurement of R&D

12. © 2013 Deloitte
12
Using Administrative Data Sources to Identify Public Procurement of R&D
Contact information
Do not hesitate to contact me for any input, questions or comments:
Lionel Kapff
Consultant – Deloitte Consulting
Mobile + 32 475 79 24 76
lkapff@deloitte.com

13. © 2013 Deloitte
13
Using Administrative Data Sources to Identify Public Procurement of R&D
Annex: Recommended readings
Pre-commercial procurement (PCP) and public procurement of innovation (PPI):
•European Commission website on PCP: http://cordis.europa.eu/fp7/ict/pcp/policy_en.html
•European Commission website on PPI: http://ec.europa.eu/enterprise/policies/innovation/policy/lead-market- initiative/public-proc_en.htm
•European Commission (2007): Pre-commercial Procurement: Driving innovation to ensure sustainable high quality public services in Europe, COM(2007) 799 final.
•European Commission (2005): Public Procurement for research and innovation.
•European Parliament (2009): Pre-commercial procurement: driving innovation to ensure sustainable high quality public services in Europe.
•ECORYS (2011): Study on pre-commercial procurement in the field of Security.
•ERAPRISM (2010): Public Procurement for Innovation in Small European Countries.
•Independant Expert Group (2006): Pre-commercial procurement of innovation.
•OECD (2011): Demand-side innovation policies.
•OMC PTP (2009): Exploring Public Procurement as a Strategic Innovation Policy Mix Instrument.
•Ramboll (2008): Opportunities for Public Technology Procurement in the ICT-related sectors in Europe. Availability of public procurement data across Europe:
•Graux & Kronenburg (2012): State of Play: Re-use of Public Procurement Data.
•Cattaneo (2012): The e-Procurement Landscape in the EU, interim results of the “Study on e-Procurement measurement and benchmarking”.
•Siemens & Time.lex (2010): Study on the evaluation of the Action Plan for the implementation of the legal framework for electronic procurement (Phase II). OECD statistical guidelines:
•OECD (2002): Frascati Manual - Proposed Standard Practice for Surveys on Research and Experimental Development.
•OECD (2011): Guide to measuring the Information Society.

14. © 2013 Deloitte
Deloitte refers to one or more of Deloitte Touche Tohmatsu Limited, a UK private company limited by guarantee, and its network of member firms, each of which is a legally separate and independent entity. Please see www.deloitte.com/about for a detailed description of the legal structure of Deloitte Touche Tohmatsu Limited and its member firms. Deloitte provides audit, tax, consulting, and financial advisory services to public and private clients spanning multiple industries. With a globally connected network of member firms in more than 150 countries, Deloitte brings world-class capabilities and deep local expertise to help clients succeed wherever they operate. Deloitte's approximately 193,000 professionals are committed to becoming the standard of excellence. This publication contains general information only, and none of Deloitte Touche Tohmatsu Limited, Deloitte Global Services Limited, Deloitte Global Services Holdings Limited, the Deloitte Touche Tohmatsu Verein, any of their member firms, or any of the foregoing’s affiliates (collectively the “Deloitte Network”) are, by means of this publication, rendering accounting, business, financial, investment, legal, tax, or other professional advice or services. This publication is not a substitute for such professional advice or services, nor should it be used as a basis for any decision or action that may affect your finances or your business. Before making any decision or taking any action that may affect your finances or your business, you should consult a qualified professional adviser. No entity in the Deloitte Network shall be responsible for any loss whatsoever sustained by any person who relies on this publication.