In early 2013, the authors were commissioned by DIICCSRTE to develop a critical interventions framework for student equity in higher education. To answer the seemingly simple question of whether we as a sector were on track in achieving our national social inclusion goals, we must review the current student equity makeup of the sector, and determine how effective our equity initiatives are. The first part of that question was relatively easy to answer. However, finding clear, rigorous evidence of program efficacy from the literature was much more difficult. In this presentation, I will discuss the critical interventions framework and the difficulties with uncovering evidence of effectiveness as opposed to the theoretical strength of an initiative, and briefly discuss how the framework might be used in the future.
1. Developing a
Critical Interventions Framework
Dr Ryan Naylor, CSHE, University of Melbourne
Curtin University is a trademark of Curtin University of Technology
CRICOS Provider Code 00301J
05/02/2014
2. Project Background
• Commissioned project for DIICCSRTE, completed by
Dr Ryan Naylor, Dr Chi Baik, Professor Richard James
• Are we on track in achieving national social inclusion
goals?
1.
2.
Where are we?
How effective are our current initiatives?
(What appears to work well? What doesn’t?)
i. Is it possible to generate a typology of equity initiatives to
allow consolidation of research evidence?
ii. Is there evidence in the literature or from HEPPP
evaluations to support their efficacy?
3. Caution!
• Some necessary simplifications had to be made in
creating the typology and fitting the literature to it
• “Intervention” is a contentious term with troubling
associations
• New coalition government changes the policy
context
4. What has had the biggest effect on equity?
• Uncapping/deregulation of volume of undergraduate
places?
• National target for low SES participation
(and associated Mission Based Compacts)?
• HEPPP funding?
• Wider societal trends in community beliefs about the value
of undertaking higher education, entry requirements,
eligibility for participation?
• Efficacy of equity initiatives depends on underlying factors
such as these
• Many variables, highly inter-related
6. Since 2007, there has been an explosion
in domestic student numbers
2007: approx. 722,000 domestic students
2011: approx. 888,000 domestic students
= An increase of 23% over 4 years, or
an annual growth rate of 5%
This level of growth is unprecedented in Australian HE
8. …But they have been relatively modest
and not universal
Participation
Ratio
(2011)
9. Average growth rate (2008-2011) (%)
Total
Total
Remote
Regional
Low SES (CD measure)
Low SES (postcode
measure)
Women in Non-Traditional
Areas
Students from a Non
English Speaking
Background
Indigenous students
Students with a disability
It is difficult to improve equity
during growth periods
12
10
8
6
4
2
0
10. Average growth rate (2008-2011) (%)
Total
Total
Remote
Regional
Low SES (CD
measure)
Low SES (postcode
measure)
Women in NonTraditional Areas
Students from a Non
English Speaking…
Indigenous students
Students with a
disability
But some groups whose share has
historically been stable have
increased their participation share
12
10
8
6
4
2
0
11. This change has not been uniform
– low SES
Access rate (2011)
(%)
Change since
2007 (%)
A
4.25
-0.01
B
6.92
2.38
C
7.10
-0.39
D
15.14
-0.14
E
20.49
-1.38
F
32.82
0.66
G
33.83
2.88
Sector average
16.87
0.75
• No correlation between access rate (2011, 2007) and change
• Complex factors – different geographical contexts, access policies,
etc
• Traditional strong performers didn’t do better
13. Total
Most groups (bar 2) are
no less likely to succeed
May not have seen full effects yet – not all students from DDS
cohort have moved through the system yet (early days yet!)
14. For most groups,
the key problem continues to be access
Equity group
Participation
Retention
Success
Students with a disability
0.48
0.96
0.93
Indigenous students
0.55
0.85
0.81
Students from Non English Speaking
Background
0.82
1.04
0.97
Women in Non-Traditional Areas
0.35
1.01
0.99
Low SES (postcode measure)
0.67
0.98
0.97
Low SES (CD measure)
0.62
0.97
0.96
Regional
0.64
0.98
0.99
Remote
0.39
0.91
0.94
Participation and access are (should be?) key focal
points in student equity and social inclusion
15. For most groups,
the key problem continues to be access
Equity group
Participation
Retention
Success
Students with a disability
0.48
0.96
0.93
Indigenous students
0.55
0.85
0.81
Students from Non English Speaking
Background
0.82
1.04
0.97
Women in Non-Traditional Areas
0.35
1.01
0.99
Low SES (postcode measure)
0.67
0.98
0.97
Low SES (CD measure)
0.62
0.97
0.96
Regional
0.64
0.98
0.99
Remote
0.39
0.91
0.94
This is not to argue they don’t need support once enrolled
• less academically well prepared students from any background
• Indigenous students
16. How effective are our current initiatives:
The Critical Interventions Framework
17. The Critical Interventions Framework
• Is it possible to create a typology of equity initiatives?
• Can we find sufficient evidence in the literature to point
to the efficacy of particular types of initiatives?
• Can we identify the initiatives that are most effective?
• We don’t know for sure, so we had to make some
guesses
18. The Critical Interventions Framework
• Is it possible to create a typology of equity initiatives?
• Yes
• Can we find sufficient evidence in the literature to point
to the efficacy of particular types of initiatives?
• Often no
• Can we identify the initiatives that are most effective?
• No. There simply isn’t enough evidence.
23. The Critical Interventions Framework
(one small section)
• We all have intuitions about what works
• Attempted to base framework on evaluative science, not intuition
• Unfortunately, the science is largely not there, so some estimation and
judgement was involved
24. All things considered,
what did we rate highly?
• Not intended to narrow or homogenise people’s efforts
High
1B. Later-year outreach (Years 10-12)
2B. Bridging/foundation programs
2D. Scholarships
4C. Student services provision
5A. Monitoring student completion rates
Very High
1D. School curriculum enhancement/support
2A. Pathway/articulation programs
2C. Alternate selection criteria and tools
3A. First year orientation/transition support
26. Do we have good evidence?
High
Quality of Evidence
1B. Later-year outreach (Years 10-12)
Limited
2B. Bridging/foundation programs
Some (from US)
2D. Scholarships
Strong (needs-based, not merit)
4C. Student services provision
Varies depending on service
5A. Monitoring student completion rates
N/A
Very High
1D. School curriculum enhancement/support
Some (from US); strong for need
2A. Pathway/articulation programs
Mixed
2C. Alternate selection criteria and tools
Strong
3A. First year orientation/transition support
Strong (from US)
30. How are we spending our HEPPP funding?
• This is purely descriptive, not normative!
• Not all initiatives require the same amount of funding
High
Proportion of HEPPP funding
1B. Later-year outreach (Years 10-12)
14
2B. Bridging/foundation programs
7
2D. Scholarships
14
4C. Student services provision
15
5A. Monitoring student completion rates
N/A
Very High
1D. School curriculum enhancement/support
1
2A. Pathway/articulation programs
3
2C. Alternate selection criteria and tools
2
3A. First year orientation/transition support
10
Total: 66%
31. Where to next?
• Written for national policy purposes – not meant to
function at the institutional level
• We need better evidence of program efficacy
(and this is where institutions come in)
o Detailed, rigorous and published evaluations
o To enable a sector-wide conversation about
equity initiatives
• How can the CIF be used in an institutional context?
Is this typology helpful?
32. Where to next?
• Funding sought to research:
o Process and attitudinal factors affecting program
evaluation
o Resources for embedding evaluation into core
business
• First year experience survey
• Collaborations with NCSEHE staff
