The OECD is a leading organization in the international measurement of health system performance. The OECD Expert Group on Health Care Quality Indicators (HCQI) has recently revised its performance framework, identifying core indicators and highlighting new directions. Although improving, the capacity of countries to deliver more accurate standardized indicators still needs to be fostered. A particular aspect that deserves attention is the design, planning and implementation of public performance reporting. Such activity, strictly interrelated to the capacity of the information infrastructure, also depends from cultural, organizational and political conditions that can be differently present at the international level. The applicability of standardized principles and the evidence of improved outcomes due to public reporting systems is still questioned to a large extent. A first international conference on the topic of hospital performance reporting has been organized in Rome, Italy in 2014, followed by a second event held in Seoul, South Korea, in 2015. In his talk, Fabrizio Carinci will present recent developments of OECD projects, including: • state of the art in the definition of OECD performance indicators • challenges emerging from OECD R&D studies • transferability and use of definitions at sub-national and provider level • applicability for hospital performance benchmarking and geographical variation • limitations imposed by the legislation on privacy and data protection • an overarching vision of “essential levels of health information” Through practical examples drawn from his direct experience as Member of the Bureau of the HCQI and other relevant Boards, Prof. Fabrizio Carinci will discuss the state of the art, the role played by national governments (including Australia), and potential avenues for mutual collaboration.

1. The revised OECD Health Systems
Performance Framework:
methodological issues and implications for
comparative public reporting at a global level
Fabrizio Carinci
Professor of Health Systems and Policy
School of Health Sciences, University of Surrey
Member of the OECD Expert Group on Health Care Quality Indicators
Forum
Agency for Clinical Innovation
Sydney, Australia
Wednesday, 24th
February 2016
f.carinci@surrey.ac.uk

2. Fabrizio Carinci
OECD Health Care Quality Indicators Project

Under the umbrella of the Organisation for Economic Cooperation
and Development (OECD), the ‘Health Care Quality Indicators’
(HCQI) Project was initiated in 2001

The general objective was to help Member States (MS) identify
priority areas for quality improvement to provide achievable
standards by examining results among best performing countries

In 2006, the OECD released a common conceptual framework for
health system performance. Nested “quality matrix” with vertical
dimensions of ‘effectiveness’, ‘patient safety’ and
‘responsiveness/patient-centeredness', horizontally subdivided
according to levels of health care needs over the life cycle:
‘staying healthy’ for healthy subjects, ‘getting better’ for people
affected by a disease, ‘living with illness or disability’ for those with
a chronic condition and ‘coping with end of life’ for terminal patients.

3. Fabrizio Carinci
Matrix Dimensions: Quality
Effectiveness
 Achieving desirable
outcomes, given
the correct
provision of
evidence-based
health care
services to all who
could benefit
 Achieving desirable
outcomes, given
the correct
provision of
evidence-based
health care
services to all who
could benefit
Safety
 System has the
right structures,
renders services
and attains results
in ways that
prevent harm to the
user, provider, or
environment
 System has the
right structures,
renders services
and attains results
in ways that
prevent harm to the
user, provider, or
environment
Responsiveness/
Patient
centeredness
 System actually
functions by
placing the
patient/user at the
center of its
delivery of health
care
 System actually
functions by
placing the
patient/user at the
center of its
delivery of health
care

4. Fabrizio Carinci
HCQI Outputs

Since 2007, results of the HCQI project have routinely
contributed to international comparisons through the
publication of the OECD series ‘Health at a Glance’ and the
release of OECD Health Statistics alongside other
international health data on expenditure, resources, utilization
and outcomes.

In 2013, the OECD HCQI data collection process included a
total of 70 indicators covering the following ‘themes’: Primary
Care (PC); Acute Care (AC); Mental Health (MH); Cancer Care
(CC); Patient Safety (PS) and Patient Experiences (PEs). The
collection reports data from 34 countries, including non-
OECD member countries eg Singapore and Latvia.

5. Fabrizio Carinci
NEW
DEFINITIONS
Revised
OECD-HCQI
Health Systems
Performance
Framework 2015

6. Fabrizio Carinci
Criteria used to select HCQIs

7. Fabrizio Carinci
Carinci F, Van Gool K, Mainz J, Veillard JH, Januel JM, Kim SM, Arispe I, and Klazinga N, Towards actionable
international comparisons of health system performance: expert revision of the OECD framework and quality
indicators, Int J Qual Health Care. 2015 Apr;27(2):137-46.

8. Fabrizio Carinci
Effectiveness – Getting better
Carinci F, Van Gool K, Mainz J, Veillard JH, Januel JM, Kim SM, Arispe I, and Klazinga N, Towards actionable
international comparisons of health system performance: expert revision of the OECD framework and quality
indicators, Int J Qual Health Care. 2015 Apr;27(2):137-46.

9. Fabrizio Carinci
Effectiveness – Chronic care
Carinci F, Van Gool K, Mainz J, Veillard JH, Januel JM, Kim SM, Arispe I, and Klazinga N, Towards actionable
international comparisons of health system performance: expert revision of the OECD framework and quality
indicators, Int J Qual Health Care. 2015 Apr;27(2):137-46.

10. Fabrizio Carinci
Safety – Primary and secondary prevention
Carinci F, Van Gool K, Mainz J, Veillard JH, Januel JM, Kim SM, Arispe I, and Klazinga N, Towards actionable
international comparisons of health system performance: expert revision of the OECD framework and quality
indicators, Int J Qual Health Care. 2015 Apr;27(2):137-46.

11. Fabrizio Carinci
Safety – Getting better
Carinci F, Van Gool K, Mainz J, Veillard JH, Januel JM, Kim SM, Arispe I, and Klazinga N, Towards actionable
international comparisons of health system performance: expert revision of the OECD framework and quality
indicators, Int J Qual Health Care. 2015 Apr;27(2):137-46.

12. Fabrizio Carinci
Responsiveness / Patient centredness
Primary and Secondary Prevention
Carinci F, Van Gool K, Mainz J, Veillard JH, Januel JM, Kim SM, Arispe I, and Klazinga N, Towards actionable
international comparisons of health system performance: expert revision of the OECD framework and quality
indicators, Int J Qual Health Care. 2015 Apr;27(2):137-46.

13. Fabrizio Carinci
HCQI 2015

14. Fabrizio Carinci
Mortality after AMI
Source: OECD Health at a Glance 2015
http://www.oecd.org/health/health-systems/health-at-a-glance-19991312.htm

15. Fabrizio Carinci
Mortality after Stroke
Source: OECD Health at a Glance 2015
http://www.oecd.org/health/health-systems/health-at-a-glance-19991312.htm

16. Fabrizio Carinci
OECD Patient Safety Indicators 2015

17. Fabrizio Carinci
Performance Dashboards: Australia
Source: OECD Health at a Glance 2015
http://www.oecd.org/health/health-systems/health-at-a-glance-19991312.htm

18. Fabrizio Carinci
Emerging international issues in public reporting

The ability to report on health systems performance is directly
related to the capacity of the information infrastructure (eg
possibility to perform data linkage at varying degrees of
complexity)

The level of implementation may also depend from specific
cultural, organizational and political conditions that can vary
both within and across countries

A range of OECD studies show that the applicability of
standardized definitions is still limited: more efforts are required

Public reporting as a cost effective solution can hardly be
demonstrated in general at regional/state/national level

19. Fabrizio Carinci

Ketelaar NA, Faber MJ, Flottorp S, Rygh LH, Deane KH, Eccles MP. Public release of performance data
in changing the behaviour of healthcare consumers, professionals or organisations. Cochrane Database
Syst Rev. 2011 Nov 9

Fung CH, Lim YW, Mattke S, Damberg C, Shekelle PG. Systematic review: the evidence that publishing
patient care performance data improves quality of care. Ann Intern Med. 2008 Jan 15;148(2):111-23.

Berwick DM, James B, Coye MJ. Connections between quality measurement and improvement, Med
Care. 2003 Jan;41(1 Suppl):I30-8.

Selection pathway

Limited evidence

Change pathway

Good evidence for hospitals

No evidence for individual
providers

Scant evidence on improved
clinical outcomes
Is Transparency evidence-based?

20. Fabrizio Carinci
Why public reporting?
So, why doing public reporting?

To implement legislation passed by Parliaments around the world
to achieve higher levels of transparency “no matter what” to
respect the citizens' rights to be informed

To achieve greater efficiency by making all public servants directly
accountable for their performance

To support pay for performance schemes that can routinely use
open benchmarking for continuous quality improvement

To provide the basis (eg hospital reporting system) for specific
solutions that can be proved to be cost effective in selected clinical
areas or for targeted population subgroups (RCTs)

To improve data quality through a continuous relation with a
prurality of users

21. Fabrizio Carinci
OECD HCQI Bureau Meetings
Rome, ITALY - Friday 26th September 2014
OPPORTUNITIES AND CHALLENGES OF
HOSPITAL PERFORMANCE PUBLIC REPORTING AT
THE NATIONAL LEVEL: INTERNATIONAL
EXPERIENCES AND FUTURE PERSPECTIVES
International workshop
organized by: Italian National Agency for Regional Health
Services (AGENAS) , Italian Ministry of Health
Progetto Mattone Internazionale
Seoul, KOREA - Thursday 17th September 2015
INTERNATIONAL SYMPOSIUM ON EXPERIENCES OF HOSPITAL
PERFORMANCE MEASUREMENT OF OECD MEMBER COUNTRIES
Organized by: Health Insurance Review and Assessment Service
(HIRA)

22. Fabrizio Carinci
Canada: Health Systems Performance Framework

23. Fabrizio Carinci
Canada: Interactive OECD tool

24. Fabrizio Carinci
Canada: benchmarking Provinces vs Canada
OECD variation of HCQI

25. Fabrizio Carinci
UK: NHS Choices

26. Fabrizio Carinci
UK: NHS Consultant choice

27. Fabrizio Carinci
UK: Funnel Plots
Risk-adjusted in-hospital mortality after elective abdominal aortic aneurysm repair: surgeon
figures in comparison to national average
Source: National Vascular Registry. 2013 Report on Surgical Outcomes. Consultant level Statistics. June 2013 c

28. Fabrizio Carinci
Italian National Outcome Program
http://95.110.213.190/PNEed14_EN/index.php
Outcome measures
by Hospital/
Local Health Unit
Audit
tools
Report Card
by Hospital/
Local Health Unit
ER Information
System
Pilot
studies

29. Fabrizio Carinci
Italy: National Outcomes Program (PNE)

30. Fabrizio Carinci
Denmark: sundhed.dk

31. Fabrizio Carinci
Denmark: Danish Quality Registries
Data entryData entry
Clinical
observations
National
audit
Regional
audit
Data
analysis
Database
Quality
improvement
Detailed feedback:
Once per year
Standard feedback:
Monthly
Results made
publically available

32. Fabrizio Carinci
Korea: decreased variation of AMI
quality score

33. Fabrizio Carinci
Dartboard diagram
To visually represent the results
of the six areas, each
organization is presented with a
specific “target” diagram, divided
into five assessment bands.
The closer the organization is to
the target level of each
performance indicator, the
nearer the relative circle will be
to the centre.
This approach is nice, but it has
known methodological
limitations and interpretative
drawbacks. There is no perfect
graphical representation!

34. Fabrizio Carinci
Exchanging best practices for fair, transparent and
effective communication to the public
PERSONAL JUDGMENT
LEAGUE TABLES SCIENCE

35. Fabrizio Carinci
Why international collaboration

To share best practices that would improve:
 understanding of the methods
 interpretation of national results and interregional variability through
common benchmarks
 understanding of data limitations and targeted strategies to
strengthen the information infrastructure and overall data quality
 uptake of effective strategies for quality and outcomes improvement

To gain efficiency through fostered international cooperation, allowing
to share costs and gain access to high level skills/tools that are difficult
to obtain at national level

To help defining the “Essential Levels of Health Information” for the
definition of international standards of performance evaluation

36. Fabrizio Carinci
Essential levels of health information in Europe
Carinci F. Health Policy. 2015 Apr;119(4):530-8.
Recently, the European
Commission coordinated
a discussion between
MS and Expert
Networks on how (and
whether) to realize a EU
health information
system.
The conceptual
framework for the
„Essential Levels of
Health Information”
originated from this
experience

37. Fabrizio Carinci
Essential levels of health information in Europe
Carinci F. Health Policy. 2015 Apr;119(4):530-8.

38. Fabrizio Carinci
Open questions

Can system-level indicators be used at provider level? How can a system-
oriented performance framework be adapted to hospitals or specific
providers? Which OECD quality indicators can be adopted for hospital
reporting?

How can we identify international benchmarks for continuous hospital
performance reporting? Which indicators shall be routinely communicated
to the public? How to share relevant data, and to which level of detail for
reliable analyses?

Which countries shall be routinely compared (EU, OECD,
universal/insurance, or any rule to define “peer countries”)?

How to communicate results to the public (best practices, new
approaches)? Is public reporting effective, sustainable and practically
convenient? Which forms of new communication and how can citizens
contribute to the process?

Lessons from the world......your turn!

39. Fabrizio Carinci
Forum
Agency for Clinical Innovation
Sydney, Australia
Wednesday, 24th
February 2016
Thanks for your attention!