Presentation by Paulo Santiago at the 10th annual meeting of the Senior Budget Officials Performance and Results Network held on 24-25 November 2014. Find more information at http://www.oecd.org/gov/budgeting

1. Performance Information in the Education
Sector
Session 4a – Performance Informed Budgeting in Practice using a
Sectoral Perspective
10th Annual Meeting of the OECD Senior Budget Officials Performance and
Results Network
Paris, 24-25 November, 2014
From Education System Evaluation to Funding in
Education
Paulo Santiago, Senior Analyst
Directorate for Education and Skills

2. Outline of Presentation
1. Expenditure in School Education
2. Education System Evaluation: Generating
Performance Information
3. The OECD School Resources Review
4. Funding of School Education

3. 1. Expenditure in School Education

4. In 2011, OECD countries spent an average of 3.9% of their GDP on
primary, secondary and post-secondary non-tertiary education
Expenditure on educational institutions as a percentage of GDP (2011). From public and
private sources, by level of education and source of funds
0
0.5
1
1.5
2
2.5
3
3.5
4
4.5
5
5.5
New Zealand
Argentina
Iceland
United Kingdom
Ireland
Denmark
Belgium
Israel
Colombia
Korea
Australia
Finland
Switzerland
Netherlands
Canada
Mexico
France
Sweden
Slovenia
United States
Chile
Portugal
Austria
EU 21 average
Poland
Estonia
Luxembourg
Spain
Germany
Italy
Latvia
Japan
Czech Republic
Slovak Republic
Russian…
Norway
Brazil
% of GDP
Primary, secondary and post-secondary non-tertiary education
Public expenditure on education institutions Private expenditure on education institutions
OECD average (total
expenditure)

5. In 2011, 13% of total public spending was devoted to education
Total public expenditure on education as a percentage of total public expenditure (1995, 2005,
2011)
0
5
10
15
20
25
New Zealand
Mexico
Brazil
Korea
Switzerland
Iceland
Denmark
Norway
Australia
Israel
Estonia
United States
Canada
Sweden
Ireland
OECD average
Belgium
Finland
EU21 average
United Kingdom
Netherlands
Austria
Poland
Slovenia
Germany
Russian Federation
Portugal
Slovak Republic
Spain
Czech Republic
France
Hungary
Japan
Italy
% of total public
expenditure 2011 2005 1995

6. Source: PISA 2012 Results: What makes schools successful? Resources, policies and practices, Volume IV, Figure IV.1.8.
Slovak Republic
Czech Republic
Estonia
Israel
Poland
Korea
Portugal
New Zealand
Canada
Germany
Spain
France
Italy
Singapore
Finland
Japan
Slovenia Ireland
Iceland
Netherlands
Sweden
Belgium
United Kingdom
Australia
Denmark
United States
Austria
Norway
Switzerland
Luxembourg
Viet Nam
Jordan
Peru
Thailand
Malaysia
Uruguay
Turkey
Colombia
Tunisia
Mexico
Montenegro
Brazil
Bulgaria
Chile
Croatia
Lithuania
Latvia
Hungary
Shanghai-China
R² = 0.01
R² = 0.37
300
350
400
450
500
550
600
650
0 20 000 40 000 60 000 80 000 100 000 120 000 140 000 160 000 180 000 200 000
Mathematics performance (score points)
Average cumulative spending per student from the age of 6 to 15 (USD, PPPs)
Above a certain level of investment, the key factor is how
to spend available funding most effectively
Student performance and average spending per student

7. 2. Education System Evaluation:
Generating Performance
Information

8. OECD educational policy reviews
OECD Reviews of Evaluation and Assessment in Education (Schools)
Highlights the importance of context. Different systems are at different
stages in developing an evaluation and assessment framework.
Expansion of evaluation in
school systems
Greater reliance on
educational standards
Increased importance of
measurement and
indicators
Accountability purpose is
gaining importance
Building capacity for
Evaluation and Assessment
(E&A)
Aligning goals of education
system with E&A
Going beyond measurement
Effectively conceiving
accountability
Main trends Policy priorities

9. Education System Evaluation:
Generating Performance Information
• Drivers:
– The rising importance of education in a global world
– The growing imperative of an efficient use of public resources
– Greater decentralisation and school autonomy
– Greater accountability in the public sector – including in budgetary procedures
– The growing importance of evidence-based policy
• Purposes:
– To monitor:
• Student outcomes at a given point in time (differences among different regions within
the education system and given student groups) and changes in student outcomes
over time
• Broader outcomes of education systems (e.g. labour market outcomes; social
outcomes)
• The impact of given policy initiatives or educational programmes
• Demographic, administrative and contextual data which are useful to explain the
outcomes of the education system
– To generate and feedback information for different agents in the education system;
– To use the generated information for analysis, development and implementation of
policies.

10. Education System Evaluation:
Reference standards
– National education goals and objectives, e.g.
• Provide high-quality education to students; promote national values
and civic responsibilities; develop skills in the economy.
– Specific priorities: e.g. improve equity; goals for specific groups
– References used in national assessments
• National curriculum goals; National learning progressions; National
standards; National curriculum goals and standards
– Specific targets set to be achieved over a certain timeline
• Also supra-national, e.g. EU benchmarks

11. Education System Evaluation:
Specific targets
Examples of targets:
– Mexico (2007-2012): education system evaluation framed by Education Sector
Programme with 6 policy objectives (e.g. promotion of ICT in education) and 41
indicators (each with a target and the respective measure)
– Northern Ireland: Programme for Government 2011-15 includes high-level targets
for the performance of the education system by 2015 (e.g. 66% of young people
achieve at least 5 General Certificates in Secondary Education with a mark of A to C
in mathematics, English and three other subjects)
– Supra-national, e.g. EU benchmarks to be achieved by 2020
• At least 95% of children between the age of 4 and the age for starting primary
education should participate in early childhood education
• The share of 15-year-olds with insufficient abilities in reading, mathematics and
science should be less than 15% (measure by PISA)
• The share of early leavers from education and training should be less than 10%
• The share of 30-34 year-olds with tertiary educational attainment should be at
least 40%

12. Education System Evaluation:
Procedures
• Instruments:
– Indicator frameworks;
– Tools to monitor student outcomes (national assessments; longitudinal
research and surveys; international assessments);
– Qualitative reviews of particular aspects of the school system (ad hoc
reviews; evaluative information generated via external school system
reviews);
– Stakeholder surveys; and
– The evaluation of specific programmes and policies

13. Education System Evaluation:
Prominence of international student assessments
• The profile of the results from international student assessments has been
significantly raised in national policy discussions
– Perceived as indicators of future economic competitiveness
– Highlighted the importance of monitoring student outcomes
– Aspirational targets for performance are established (e.g. in PISA)
• A number of countries has established PISA targets
– Denmark (2010): Danish students to be in the top five countries as judged in
international assessments.
– Australia (National Plan for School Improvement, 2012): to be among the top five
school systems in the world by 2025 in mathematics, science and reading
achievement.

14. Student performance and equity
Peru
Chile
Bulgaria
Hungary
Slovak Republic
Portugal
Luxembourg
France
Uruguay
New Zealand
Chinese Taipei
Belgium
Costa Rica
Romania
Israel
Germany
Indonesia
Colombia
Tunisia
Argentina
Brazil
Malaysia
Turkey
Greece
Lithuania
Latvia Russian Fed. Spain
UK
Czech Republic
Denmark
Slovenia
Ireland
Austria
Viet Nam
Switzerland
Singapore
Shanghai-China
Poland
United States
Croatia
Netherlands
Montenegro
Serbia
Hong Kong-China
Estonia
Finland
Thailand
Japan
Sweden
Australia
Canada
Jordan
Macao-China
U.A.E. Kazakhstan
Iceland
Qatar
Norway
Mexico
Liechtenstein
Korea
Italy
300
350
400
450
500
550
600
650
30 25 20 15 10 5 0
OECD
average
OECD average
Percentage of variance in performance explained by ESCS (r-squared x 100)
Mean mathematics performance
Below average mathematics performance
Below average impact of socio-economic
background
Above average mathematics performance
Below average impact of socio-economic
background
Below average mathematics performance
Above average impact of socio-economic
background
Above average mathematics performance
Above average impact of socio-economic
background
Relationship between mathematics performance and variation in
performance explained by students’ socio-economic status

15. Education System Evaluation - Example
Australia
References and standards (National level)
• Council of Australian Governments (COAG) National Productivity Agenda
– Includes a set of aspirations, outcomes, progress measures, and policy directions for
education
• Melbourne Declaration on Educational Goals for Young Australians
– Articulates future directions and aspirations for Australian schooling
• Australian Curriculum
• Priority areas / Education targets [e.g. Increased proportion of young Australians attaining
secondary education; halve the gap for indigenous students within a decade]
Methods and instruments (National level)
• National Assessment Program
– NAPLAN – Literacy and Numeracy: full cohort tests in reading, writing, spelling, grammar and
punctuation, and numeracy at Year levels 3, 5, 7 and 9.
– Sample assessments: Cyclical sample surveys in science, ICT, civics and citizenship in Years 6
and 10.
– International assessments: PISA, TIMSS, PIRLS
• Measurement Framework for National Key Performance Measures
• Independent reviews

16. OECD Indicators of Education Systems
(INES) – Education at a Glance
The INES organising framework includes three major policy perspectives:
• quality of educational outcomes and educational provision;
• equality of educational outcomes and equity in educational
opportunities; and
• adequacy, effectiveness and efficiency of resource management.
Comparative
information
for different levels of
education on:
Student enrolment,
entrance and
graduation rates
Educational
personnel
Educational
finance
Comparative
information:
Teaching and
learning
environments
(TALIS 2008;
PISA surveys)
Student outcomes at age
15
Reading, Mathematics and
Science
(PISA 2000, 2003, 2006,
2009, 2012….)
Survey of Adult Skills
(16-65)
(Literacy, Problem-Solving,
Numeracy, Skills-Use)
(PIAAC)

17. Education System Evaluation:
Policy Options
Governance: Being systematic and strategic for better informed policy
making
• Ensure a broad concept of education system evaluation within the E&A framework
• Ensure policy making is informed by high-quality measures, but not driven by their
availability
• Situate education system evaluation in the broader context of public sector performance
requirements
Procedures: Developing an approach to learn from a broad evidence base
• Develop a national education indicator framework
• Design a national strategy to monitor student learning standards
• Ensure the collection of: qualitative information; and contextual information to monitor
equity
• Assure the monitoring of changes over time and progress of particular student cohorts

18. 3. OECD School Resources Review

19. OECD School Resources Review:
Objectives
Overarching policy question:
“What policies best ensure that school resources are effectively used to
improve student outcomes?”
• Key issues for analysis
o Funding of school education [including budgeting in education]
Level of resources; sources of revenue; education budgeting procedures; forecasting
resource needs; resource strategy; distribution of funding across administrative levels,
education levels, sectors and individual schools; monitoring of resource use; capacity for
resource management; transparency and reporting.
o Management of human resources (e.g. teachers, school directors)
o Organisation of school network (location, size of schools)
The Review will:
• Synthesise research-based evidence on effective resource use in the school sector
and disseminate this knowledge among countries
• Identify innovative and successful policy initiatives and practices
• Identify policy options for policy makers to consider.

20. Context
Economic
Demographic
Political
Cultural
Governance
OECD School Resources Review:
Analytical framework and Key Issues
Optimal outputs
Access,
participation,
completion
Learning outcomes
Labour market &
social outcomes
Resource
Distribution
Resource
Utilisation
Resource
Management
Education System Goals

21. OECD School Resources Review:
Participation
• Austria
• Belgium (Flemish Community)
• Chile
• Czech Republic
• Denmark
• Estonia
• Kazakhstan
• Lithuania
• Slovak Republic
• Uruguay
Country Reviews
• Belgium (French Community)
• Iceland
• Luxembourg
• Spain
• Sweden
Country
Background
Reports

22. 4. Funding of School Education

23. Funding of School Education
A diversity of
channels for
funding distribution:
Central
government to line
ministries
From Ministry of
Education to local
authorities
To schools
through funding
formulas
A range of factors used in Funding Formulae, including:
• Number of students, class size, characteristics of student population,
qualifications of teachers
• Type of educational offer (e.g. VET vs General programmes), school
location, size of school
Performance measures typically not used
Requires high-quality and reliable data at the school level

24. • Comparative indicators of school performance can miss important aspects of
school quality
– Standardised performance measures
• Often limited to discrete areas of student learning objectives
• Often cross-sectional and do not allow monitoring of school progress
• Often do not identify impact of the school on student outcomes (value-added
measures are needed)
• Use of performance measures for accountability:
• Not strongly linked to financial rewards/sanctions
• Very different policies on how these feed into decisions on possible school
closure
• Closer supervision of schools with quality concerns
• Reward of more autonomy to schools with good evaluation results
• Care with accountability uses of school performance measures
• Over emphasis on what is assessed in performance measures; narrowing of
the curriculum; teaching to the test; hinder innovation.
• Stigmatisation of particular schools and unintended impacts on parental
choice of schools. Complacency of schools performing well on such
measures.
School Evaluation: Use of performance
measures

25. Thank you for your attention!
www.oecd.org/edu/evaluationpolicy
www.oecd.org/edu/school/schoolresourcesreview.htm
paulo.santiago@oecd.org