3. • Objective: test whether it is possible to
– measure what undergraduate students know and
can do upon graduation …
– … across diverse countries, languages, cultures
and types of institutions
• For
– Generic skills
– Economics
– Engineering
Assessing Higher Education Learning
Outcomes (AHELO) Feasibility Study (2008-12)
4. •
•
Observers
Bahrein
Brazil
Saudi Arabia
Singapore
•
•
••
••
•
•
• •
•
••
•
•
•
•
•
•
•
•
•
A range of geographic, linguistic and cultural
backgrounds involved
•
Generic Skills
Colombia
Egypt
Finland
Korea
Kuwait
Mexico
Norway
Slovak Republic
United States
(CT, MO, PA)
Engineering
Abu Dhabi
Australia
Canada (Ontario)
Colombia
Egypt
Japan
Mexico
Russian Fed.
Slovak Republic
Economics
Belgium (Fl.)
Egypt
Italy
Mexico
Netherlands
Russian Fed.
Slovak Republic
5. • Data collection from February to June 2012
• 17 countries involved in 25 strand replications
• Data collected from over
– 23,000 students
– 4,900 faculties
– 270 institution coordinators
• 1,000 test sessions and 20,000 computers involved
• Scoring completed in June 2012
• Analysis of results and findings
• Final reporting 2012-13
Quick facts on AHELO Feasibility Study
6. • The main conclusions of the meeting were:
– The revised proposal should frame the assessment of
higher education learning outcomes around transversal
skills, and build any disciplinary assessments and contexts
around those.
– Individual student results and feedback will be critical to
institutional and student engagement and, therefore, the
assessment should extend to all eligible students of
participating entities, rather than a sample only.
– To report results from the assessment both in absolute
terms (bottom-line results) as well as relative to the socio-
economic and institutional context of individuals and
universities (analytic value added).
AHELO Main Study – London meeting with
interested countries on 11 Feb 2015
6
7. • The main conclusions of the meeting were:
– To devote greater attention to the development of
intermediate outputs (frameworks, instruments and
methodologies), not only to ensure that participating entities
obtain short-term value for their investments, but also to
facilitate the important dialogue with the academic
community and other stakeholders.
– The AHELO Main Study should be governed by national or
subnational government authorities, but the governing body
should be complemented with a strong Academic Advisory
Group from among participating institutions that would
oversee much of the substantive development of the
assessment. The governing body would be made up of
those countries or subnational entities that contribute to the
development and financing of the project, while recognising
that not all participating entities may be able to implement
the assessment from the outset.
AHELO Main Study – London meeting with
interested countries on 11 Feb 2015
7
8. • The AHELO Scoping Paper has been revised
accordingly and published as
EDU/EDPC(2013)17/REV3
• A new funding scheme has been developed.
• Countries are invited to express their willingness
to participate in Year 1 (2016) of the AHELO Main
Study on the basis of this proposal and to do so
before the end of May 2015.
AHELO Main Study – where are we now
8
10. • Higher education is a rapidly expanding and
globalising system with extremely low levels of
systemic transparency
• Lack of internally generated transparency leads to
various forms of external mechanisms, most
importantly rankings and consumerist approaches to
quality
– Often not providing right incentives to improve
teaching and learning
• Various national attempts to measure learning
outcomes in higher education, but lack of
international, comparative approach ≈ PISA
Background
11. 1. Globalisation of demand: what are students
actually buying?
2. Outcomes and impact: what do employers and
wider society get?
3. Doubts on the value of HE qualifications:
selection, credentialism or learning gain?
4. Growing dissatisfaction with the way rankings
are ‘ordering’ the HE system and distribute
rewards and incentives
Drivers
12. • Explosion of global demand in emerging
countries
– In what exactly are the emerging middle classes
investing enormous resources?
• Global imbalance between demand and supply
of supposed academic excellence
– Imbalance which cannot be met by international
mobility (4.5m now, expected to increase to 7.2m
in 2020) and e-learning/MOOCs
1. Globalisation
13. Global expansion & redistribution of qualifications
Global distribution of tertiary educated 25-34 y-olds in 2013 and 2030
14. 14
United
States,
13.7%
China,
17.8%
Russian
Federation,
10.9%
Japan, 6.9%
India, 11.4%
Korea, 3.9%
Mexico,
3.0%France,
2.6%
Germany,
2.0%
United
Kingdom,
2.9%
Indonesia,
4.3%
Spain, 2.2%
Canada,
2.1%
Brazil,
3.0%
Turkey,
1.7%
Other,
11.7%
Share in academic graduates 2010
United States
43.2%
United
Kingdom
13.8%
Netherlands
6.0%Germany
4.3%
Canada
4.3%
Australia
4.3%
Switzerland
3.5%
France
3.0%
Japan
2.5%
Sweden
2.6%
Korea
2.2%
Hong
Kong
2.0%
Other
8.4%
Share in academic excellence
THEWUR 2012
Global distribution of academic
graduates and academic excellence
15. • Economic outcomes
– Employment
– Earnings
• Social outcomes
– Social capital, interpersonal trust
– Health
– Political participation
– Volunteering
– Anti-social behaviour
2. Persistent high impact of HE on economic
and social outcomes
16. Employment benefits very significant
Employment rates among 25-64 year-olds, by educational
attainment (2012)
20
30
40
50
60
70
80
90
100
Iceland
Norway
Switzerland
Sweden
Germany
Netherlands
Austria
Denmark
Latvia
Brazil
Slovenia
Israel
Luxembourg
Poland
Belgium
Australia
France
Finland
Chile
UnitedKingdom
NewZealand
CzechRepublic
OECDaverage
EU21average
RussianFederation
Estonia
Portugal
Canada
Mexico
UnitedStates
SlovakRepublic
Ireland
Japan
Hungary
Italy
Spain
Korea
Turkey
Greece
%
Below upper secondary education Upper secondary or post-secondary non-tertiary education Tertiary education
17.
18. Social outcomes – interpersonal trust
Proportion of adults reporting that they trust others, by educational
attainment (2012)
0
10
20
30
40
50
60
70
SlovakRepublic
Estonia
Italy
CzechRepublic
France
Korea
Germany
Japan
Ireland
Poland
England/N.…
Average
Flanders(Belgium)
Canada
Spain
Austria
UnitedStates
Australia
Finland
Netherlands
Sweden
Norway
Denmark
%
Below upper secondary education Upper secondary or post-secondary non-tertiary education Tertiary Education
19. Social outcomes – trust in political institutions
Individuals with higher level of education more likely to believe
they have a say in government
0
10
20
30
40
50
60
70
80
CzechRepublic
Italy
Spain
Estonia
Germany
Japan
SlovakRepublic
Ireland
Canada
Korea
England/N.Ireland
(UK)
Austria
Average
Flanders(Belgium)
Australia
UnitedStates
Netherlands
Sweden
Denmark
Finland
Norway
%
Below upper secondary education
Upper secondary or post-secondary non-tertiary education
Tertiary Education
20. • What do we know about the value of higher
education’s qualification in terms of knowledge
or skills?
3. Knowledge & skills value of higher
education qualifications
23. 200 220 240 260 280 300 320 340 360 380
Spain
England (UK)
England/N. Ireland (UK)
Ireland
Italy
Korea
Canada
Poland
United States
Northern Ireland (UK)
Australia
Estonia
Average
France
Denmark
Norway
Slovak Republic
Germany
Japan
Sweden
Austria
Netherlands
Flanders (Belgium)
Czech Republic
Finland
Numeracy scores of tertiary educated adults of 25-34y old
95th percentile mean score tertiary 25-34y
24. 24
Numeracy equivalent of tertiary qualifications
Proportion of 25-64 year-olds scoring at PIAAC numeracy level
4 and 5, by educational attainment of the population (2012)
0
5
10
15
20
25
30
35
40
Japan
Finland
Netherlands
Sweden
Australia
Norway
Flanders(Belgium)
England(UK)
England/N.Ireland
(UK)
UnitedStates
CzechRepublic
OECDaverage
Poland
Canada
NorthernIreland(UK)
Austria
Germany
Ireland
France
Denmark
Estonia
SlovakRepublic
Korea
RussianFederation
Spain
Italy
Below upper secondary education Upper secondary or post-secondary non-tertiary education Tertiary education
26. • Using the THEWUR database on the Top 200
universities, how dynamic is the HE system?
– Teaching – research – citations
• Current input and reputation measures do not
provide the right incentives for improving or
innovating teaching & learning
4. Structure and hierarchy in HE and the
associated incentives system
27. Institutional moves in the THEWUR
Absolute moves
0
5
10
15
1 51 101 151
Teaching Research Citations International
Poly. (Teaching) Poly. (Research) Poly. (Citations)
28. Where are the top20 performers?
0%
25%
50%
75%
100%
0
5
10
15
20
Teaching Research Citations International Relative share of top performing uni (any scale) per country (right axis)
29. • Is it possible?
– To improve our understanding of what students actually
‘learn’ in higher education
– To exchange reputations with empirically grounded
observations of quality of teaching & learning
– To gradually transform the field on which credentials are
traded into a more level playing field
– To provide better information to students and employers
about the quality of teaching & learning experiences
– To develop feedback loops to improve teaching and learning
– To reward and incentivise institutions that significantly
improve their teaching & learning environments
– To re-confirm the value of teaching as part of the university’s
mission next to research
Conclusion: what are the value-propositions?
31. • Strong preference among interested countries
for generic skills
– Generic academic skills are at the core of the
mission of universities
– Specialised, disciplinary knowledge and skills is
more different
– Overlap with established systems or practices on
disciplinary level
– Strong preference among employers as well
Disciplines or generic skills
32. • How to respect and value institutional heterogeneity
and diversification?
• Developing various scales?
– Research skills
– Complex reasoning
– Creative thinking
– Problem-solving
– Social and emotional skills
– Communication skills
– Employability and entrepreneurship skills
– Etc…
Various scales
33. • Orientation towards assessment with clear
benefits for students
• Countries want a census-type of assessment
• Necessity to provide something of value to
universities
– Balancing accountability and improvement
functions
Who benefits: student, institution, country?
34. • Real value-added assessment considered to be
the best option, but at the same time unrealistic
• Two approaches
– Absolute scores at individual, institutional and
country level
– Analytical value-added scores based on
regressions to background variables at
institutional and country level
Absolute or value-added
