Dr Erin Wilson - School of Health and Social Development, Deakin University

Are we making a difference?
Measuring the outcomes of disability
related services and supports
Presenter: Dr Erin Wilson
School of Health and Social Development, Deakin
University
Acknowledgements: Dr Nick Hagiliassis and
Dr Robert Campain, Scope

Background- what do I bring?
• Worked with Scope; Perth Home Care Services; WA Mental Health
Commission and others
• Worked with team of researchers to develop a range of data
collection and outcome measurement tools in a range of contexts:
– Individualised funding and planning
– Early Childhood Intervention Services
– Flexible funding in early childhood
– Therapy and psychology services
– Community based mental health services
– Assistive technology
– Range of disability service models

Selection of data collection and analysis
tools developed and trialed
Scope Outcomes Framework (used for analysis)
• A whole of life outcomes framework across seven life domains
Scope Outcomes and Impacts tool
• A one page survey rating scale to measure impact (positive and negative) of service on life
domains of individual or family
Individualised Planning/Review Tool
• A tool based on the categories of a person centred plan to rate achievement against goals
Measurement of Services and Supports (MOSS) tool
• A pre and post measure of change resulting from individual goal setting and services based on
this goal
Family Stress and Coping tool AND Family Capacity tool
• Two survey scales rating a family’s self perceived level of coping and capacity
Early Childhood Individualised Support Plan tool
• Attachment to Individualised Support Plan to rate outcomes at end of service period, and
barriers and enablers to these
1 in 4 poll Social Inclusion Survey
• A population level survey rating levels of social inclusion related to service exclusion, economic
exclusion and social exclusion

Overview
• Key terms
• WHAT outcomes might we use – selecting and
defining outcomes
• HOW to measure outcomes – guiding
principles

How do we know if we’re making a
difference?
3 key ideas/terms
•Process
•Output
•Outcome

Terminology – key elements
What are ‘outcomes’?
In broadest terms, ‘outcome’ means: what happened
BUT different things are focused on and there is confusion over
terms and their relationship:
1. PROCESS: did we receive or deliver a quality service? The ways in
which program services and goods are provided. Practices used.
2. OUTPUT: how much service did we receive or deliver, how many
people received it etc? The goods and services produced (quantity,
efficiency: e.g., number of people served, speed of response to
complaints).
3. OUTCOME: what happened (for the person)? The broader results
achieved through the provision of goods and services (Horsch, 2005)

• Under the proposed NDIS, it is possible that measurement will occur across
outcomes, but also outputs and processes.
– e.g., outputs - number of clients serviced, respite hours, equipment payments,
home and vehicle modification data;
– e.g., process – degree of service coordination, how disputes are managed.
• Measuring process or output is frequently used as a ‘proxy’ measure of
outcome
• This is based on the thinking (and sometimes research) that where a
given process is used (or level of output occurs) this leads to the
desired outcome (eg. Support to exercise leads to better health –
measure whether support to exercise provided)
• Department of Human Services Standards – largely oriented to
measure ‘good practice’ i.e process
• Output or Process measures ≠ Outcomes.
– i.e., They do not reveal what happened as a result of the service or
intervention or activity.

Steps in developing an outcomes measurement
approach for human services
1. Decide the desired areas of change
– i.e desired outcomes and for whom
2. Define these areas & unpack complex terms
– eg. what do we mean by ‘communication competence’, ‘awareness’,
‘community inclusion’, ‘support’, ‘partnership’?
3. Identify the indicators of these changes
– how will we know, what will we see if change occurs?
4. Decide how these indicators can be measured
– How to collect data. How to analyse data. Do measures exist? Can existing
measures be modified? Do measures need to be developed?
5. Combine and present the resulting information in a clear and
informative way. (Adapted from ABS, 2001, ch 1).
1. Decide
the desired
areas
2. Define these
areas &
unpack
complex terms
3. Identify
the
indicators
4. Design
how to
measure
5. Present
information

Steps of outcomes measurement Task
1. Decide the desired areas of change
(what are the outcomes?)
Check if specific outcome areas are
identified by funder, government policy,
or by others such as individuals who set
specific goals for services. Consult
stakeholders (practitioners and service
recipients)
2. Define these areas & unpack complex
terms
Check published literature for concept
definition and sub domains of concepts.
Consult stakeholders
3. Identify the indicators of these changes Check published literature to identify
indicators; consult with stakeholders
4. Decide how these indicators can be
measured
Check published literature for existing
data collection instruments and methods;
consult stakeholders; design methods
5. Combine and present the resulting
information
Report and publication writing

Step 1: decide the desired areas –YY
Which processes should we measure?
Processes / practice approaches we might measure:
– Family centred practice
• (Process measurement tool is Measure of Processes of Care (MPOC), King,
Rosenbaum & King, 1995)
– Person centred practice
– Individualised service delivery
– Strengths based practice
– Recovery based practice
– Etc, etc

Steps 1,2, 3:
Decide desired areas; Define these areas; Identify indicators.
An example: Family Centred Practice
1. Decide
the desired
areas
2. Define
these areas
& unpack
complex
terms
3. Identify
the
indicators
Family
Centred
Practice
5 Domains:
1. Enabling and
Partnership
2. Providing general
information
3. Providing specific
information
4. Co-ordinated and
comprehensive care
5. Respectful and
supportive care
(King, Rosenbaum and King, 1995)
Enabling and Partnership e.g.,
Parents felt like partners in their
child’s care
Providing general information e.g.,
Parents received adequate
information about their child’s
disability
Co-ordinated and comprehensive
care e.g., Looking at the needs of the
‘whole’ child (e.g. Mental, emotional
and social needs) instead of just
physical needs

Steps 1,2, 3:
An example: Department of Human Services Standards
1. Decide
the
desired
areas
2. Define
these areas
& unpack
complex
terms
3. Identify
the
indicators
Department
of Human
Services
Standards
(VIC)
4 Areas:
1. Empowerment
2. Access and
Engagement
3. Wellbeing
4. Participation
(DHS, 2012)
Empowerment e.g.,
Rights and responsibilities are developed and
provided in an accessible format that
facilitates understanding by all people
Access and Engagement e.g.,
Service-delivery hours are responsive to the
needs of people accessing the service
Wellbeing e.g., People actively participate in
an assessment of their strengths, risks, wants
and needs
Participation e.g., The service provider
supports people to develop and maintain their
personal, gender, sexual, cultural, religious and
spiritual identity

Steps 1,2, 3:
An example: Person Centred Practice
1. Decide
the desired
areas
2. Define
these areas
& unpack
complex
terms
3. Identify
the
indicators
Person
Centred
Practice
Domains / major areas:
??
??

Step 1: decide the desired areas of change –
yy
Which outcomes should we measure?
Outcomes for whom?
• Person with a disability
• Family /carer of person
• Service provider
• Service sector
• Community members/organisations
• Wider society and social structures
Frequently
capacity
building
and social
change
outcomes

Step 1: decide the desired areas of change –
yy
• What outcomes were seen as
important by panel?
• What others would you add?
• What are the most important?

Step 1: decide the desired areas of change -
• Outcomes for persons with disability might include:
– Quality of life
– Social inclusion
– Participation
• (community participation, cultural participation, social
participation, political participation, economic participation….)
– Satisfaction (with services, with life…)
– Loneliness
– Autonomy, independence, choice….
– Etc, etc

What are the outcomes sought under
the NDIS?
“An important component of the evidence base under the
NDIS will be the systematic collection of data on outcomes
of particular services or interventions for people with
disabilities. Data on outcomes in employment, education,
social participation, and capacity for self-care, and on the
measures that contributed to those outcomes, would help
to build an evidence base for analysing which interventions
or forms of assistance are more effective, and why”.
p 576, Productivity Commission Report, Disability Care and Support, 2011

What are the outcomes sought in other important
policy documents and Conventions?
Convention on the Rights of Persons with Disabilities (UN, 2006):
• Full and effective participation of all people with disabilities in all areas of community
and economic life
• equality, autonomy, independence, opportunity, accessibility, identity, participation
and inclusion.
National Disability Strategy:
• Inclusive and accessible communities
– Proportion of people using public transport; proportion participating in common cultural and
recreational activities
• Rights, protection, justice and legislation
– Feelings of safety; participation in civic life
• Economic security
– Average income; housing stress; participation in labour market
• Personal and community support
– Access to and unmet demand for disability services, etc
• Learning and Skills
• Health and Wellbeing

Step 2: Define these areas & unpack complex
terms
– eg. what do we mean by ‘social inclusion’
Step 3. Identify the indicators of these changes
– how will we know, what will we see if change
occurs?
2. Define
these areas
& unpack
complex
terms
3. Identify
the
indicators

Steps 1,2, 3:
An example: Social Inclusion (as used in 1 in 4 poll)
1. Decide
the desired
areas of
change
2. Define
these areas
& unpack
complex
terms
3. Identify
the
indicators
Social
inclusion
Social exclusion
Service exclusion
Economic exclusion
(Saunders et al 2007a,
2007b & 2008)
Social exclusion e.g.,
Having social contact with
other people
Service exclusion e.g.,
Accessing medical services
Economic exclusion e.g.,
Having enough money to get
by on

The 1 in 4 Poll - Social Inclusion Survey
(example items from on line and paper based survey in Easy English)
4. Design
how to
measure

Steps 1,2, 3:
An example: Quality of life
1. Decide
the desired
areas of
change
2. Define
these areas
& unpack
complex
terms
3. Identify
the
indicators
Quality
of Life
1. Emotional wellbeing
2. Interpersonal
relations
3. Material wellbeing
4. Personal
development
5. Physical wellbeing
6. Self determination
7. Social inclusion
8. Rights
(Schalock, 2004)
Emotional wellbeing e.g.,
Contentment (moods, enjoyment);
Self concept (eg self esteem)
Material wellbeing e.g.,
Financial status (income, benefits);
Employment; Housing
Physical wellbeing e.g., Health;
Activities of daily living; leisure

Steps 1,2, 3:
An example: Outcomes Star (Services Connect)
1. Decide
the desired
areas of
change
2. Define
these areas
& unpack
complex
terms
3. Identify
the
indicators
Outcomes
Star
Services
Connect
1. Motivation and taking responsibility
2. Self care and living skills
3. Managing money and personal
administration
4. Social networks and relationships
5. Drug and alcohol misuse
6. Physical health
7. Emotional and mental health
8. Meaningful use of time
9. Managing tenancy and
accommodation
10. Offending (Triangle, n.d)

Reflections on WHAT outcomes to
measure: Guiding principles
1. Outcomes selected must be meaningful
• Must capture what is important to people most
affected.
– Consider whole of life rather than focus on single aspects
– Consider broad outcomes list to include ALL that is
important to people
– Consider allowing individuals to name own outcomes (eg
goal focused) – able to be configured to the individualised
service and support context.

Example: A Whole Of Life, Whole Of Person Approach
Scope Outcomes Framework (Wilson, 2006)
• Encapsulates nine life areas:
Personal, Social, Economic, Political, Educational, Recreation and Leisure,
Cultural, Spiritual, Environment
• Reflects
– multidimensional aspects of a person’s life
– move away from outcomes measurement approaches that have a narrower
focus, e.g., those that focus on physical functioning exclusively.
– whole of life/person approach (not just the piece of it the service addresses).
• Drawn from a wide range of literature (and concepts) such as:
– Personal Outcomes Measures (Accreditation Ontario, 2000),
– National Core Indicators (Human Service Institute, 2003),
– Quality of life and subjective well being concepts (e.g., Cummins, 2007; Shalock,
2004; International Wellbeing Group,2005),
– wide range of other literature (happiness, friendship, social capital, etc),
– lived experiences of clients, families and communities involved in Scope
services.

Scope Outcomes Framework / Life domains model
(Wilson, 2006)
Personal and family
wellbeing
That is, in the area of health and function, happiness, wellbeing, safety, sense of
independence and choice.
Social life That is, in the area of friendship and relationship, community involvement and sense
of belonging
Political life That is, in the area of having a say about things that affect you (eg in a local service
or community group, about your area, school, funding etc
Cultural life That is, being involved in cultural activities (eg. arts, music, theatre, dance at any
level). This might be through attending activities or playing an active part. Or being
part of your own cultural group
Recreational and
leisure life
That is, being involved in recreational or leisure activities at any level. This might be
through attending activities, playing an active part or doing what you enjoy
Economic life That is in terms of your finances, employment or business.
Educational life That is, any aspect of your family’s education, training, personal or professional
development.
Spiritual life That is, any aspect of your family’s religious or spiritual activities
Your environment
That is, your family’s access to and enjoyment of public spaces (eg parks, pools,
theatres, shopping centres, public transport etc) or your own private space (eg. your
home). Making environments you use more accessible and appropriate (eg
playgroups, kindergartens, library, etc).

Outcomes across whole of life focus: Outcomes and Impacts tool based on
the life domains model (Wilson, 2006)

Thinking about whole of life or breadth of outcome coverage
Scope/Wilson life domains Personal Outcome Measures (Council on Quality Leadership, 2012)
Personal and family wellbeing
People are safe
People have the best possible health
People exercise rights
People are treated fairly
People are free from abuse and neglect
People experience continuity and security
People decide when to share personal information
People choose where and with whom they live
People choose services
People choose personal goals
People realize personal goals
Social life People are connected to natural support networks
People have intimate relationships
People interact with other members of the community
People perform different social roles
People participate in the life of the community
People have friends
People are respected
Political life
Cultural life
Recreational and leisure life
Economic life People choose where they work
Educational life
Spiritual life
Your environment
People use their environments
People live in integrated environments

Reflections on WHAT outcomes to measure:
Guiding principles continued
2. Information collected should be able to guide
improvement
– “The point is not to collect information - it is to learn enough
about how things are going so we can keep going or change
direction with confidence. We want monitoring which leads to
action to improved… [practice], and to have a positive impact on
people’s lives” (Ritchie et al, 2001: 177)
– Identification of factors affecting outcomes: enablers, barriers.

2. What worked for you? What helped you achieve your goal?
• People, activities, equipment/aids/resources, finances, environment, policies, time. What else?
• Which of these was the biggest help?
Comments from person with a disability
Comments from others (please specify role, eg. paid support worker, family):
Practitioner comments:
3. What stopped you from achieving your goal? What made achieving your goal difficult?
• Which of these was the biggest barrier?
Comments from person with a disability:
Example from MOSS: Enablers and Barriers focus

Step 4: Design HOW these indicators
can be measured
Guiding principles for HOW to measure:
1. Self report
2. Achievable
3. Accessible
4. Effective
See also Andresen, 2000
4. Design how
to measure

HOW to measure: Guiding principles
1. Self report of person with disability (or those most
affected, e.g. carer)
– Capture the view point of a person with a disability
– Create alternative ways a person can give information directly
about themselves
• E.g. Use of multi media
• Provide a scribe to assist write, type or enter data
– If must use proxy to report, give explicit instruction to report
from the viewpoint of person they represent
– Be clear who is providing data
• Identify different respondent views – distinguish between person
with disability (service recipient), staff, and others
– Have a mechanism for informed consent of the person

Examples of self report
• Ensuring the perspective of the person with a disability is
captured – explicit instruction to proxies;
– 1 in 4 poll:
67.4% of respondents completed the survey independently;
15% had carers complete the survey on their behalf.

The survey is being completed by:
 A person with disability without assistance
 A person with a disability with assistance
 A carer of person with disability (someone speaking
on behalf of a person with a disability who cannot
express their views themselves)
Where support is provided, the survey should be completed from the
person with a disability’s view point

Examples of self report
• Ensuring the perspective
of the person with a
disability is captured;
– MOSS:
offers three rating scales in
different formats to enable
individuals to select which
one is meaningful to them.
Accessible for some people
with intellectual disability

2. What worked for you? What helped you achieve your goal?
• Which of these was the biggest help?
Comments from person with a disability
3. What stopped you from achieving your goal? What made achieving your goal difficult?
• Which of these was the biggest barrier?
Comments from person with a disability:
Example from MOSS: Different viewpoints

2. Achievable
– Brief and easy to use
– Avoids administrative burden
– Considers time, workload, training, data analysis
allocations
– Considers BOTH how data will be collected AND
how data will analysed

3. Accessible
– Instrument can be used (with and without support by others) by
people with a wide range of disabilities including people with
intellectual disabilities;
– Has alternative formats:
• Easy English format and/or visual supports;
• simple visual scaling;
• physical administration demands low;
• allows for simple verbal responses (e.g., ‘yes’ or ‘no’);
– Considerations to item/instrument bias; reliability, validity vs
internal inconsistency; acquiescence;
– Culture/ language adaptations.

Examples of accessibility
• Value in engaging Easy English experts to
translate content
• Checked by people with intellectual disability
E.g. 1 in 4 poll:
High ‘user friendliness’ ratings (mean rating 8.5)

Examples of accessibility
MOSS tool
• offer simplified choices and rating scales

4. Effective
– Items well match the outcome concepts identified (concept capture
and validity)
– Indices of change/measurement are responsive – sensitive to and
able to capture small changes
– Considers timing of data collection – when will outcomes be
achieved and apparent? (at moment of intervention, long term)
– Has credibility: indicators provide sufficient information about a
condition or result to convince both ‘supporters’ and ‘sceptics’
(Andresen, 2000)
– Allows for aggregating up for generating comparative data, e.g.,
comparison to general population, benchmarking data
• Data can be analysed at the individual, service provider, disability system and
societal level.

Deciding the ‘when’ of outcomes: outcomes
at what moment?
– e.g., outcomes for the person
with a disability

Example: Deciding the ‘when’
Outcomes for the person with complex communication
needs
– at the moment of intervention (e.g. use of communication
board in a shop)
• can communicate effectively
• able to make an effective transaction
• increased confidence
– beyond the moment of intervention - ie on person’s life as a
whole
• increased ability to shop independently (decrease support costs)
• increased choice and range of places can participate in

Issues we’ve confronted
• Selecting the ‘right’ indices of measurement (e.g.
performance, achievement, satisfaction, impact, difficulty,
etc)
• Aggregating data – moving from a focus on an individual’s
outcomes (highly individualised) to whole of service /
population outcomes
• Data analysis – difficulties in quantifiers, thematising etc –
time consuming and often highly subjective
• Administration compliance – getting staff to administer
data collection tools (correctly)
• And more…..

Data collection and analysis tools
developed and trialed
Scope Outcomes Framework (used for analysis)
• A whole of life outcomes framework across seven life domains
Scope Outcomes and Impacts tool
• A one page survey rating scale to measure impact (positive and negative) of service on life
domains of individual or family
Individualised Planning/Review Tool
• A tool based on the categories of a person centred plan to rate achievement against goals
Measurement of Services and Supports (MOSS) tool
• A pre and post measure of change resulting from individual goal setting and services based on
this goal
Family Stress and Coping tool AND Family Capacity tool
• Two survey scales rating a family’s self perceived level of coping and capacity
Early Childhood Individualised Support Plan tool
• Attachment to Individualised Support Plan to rate outcomes at end of service period, and
barriers and enablers to these
1 in 4 poll Social Inclusion Survey
• A population level survey rating levels of social inclusion related to service exclusion, economic
exclusion and social exclusion

Caution
Outcome measurement is a not an exact science – it tells us some
things but not everything, only part of the story
Not everything that can be counted counts,
and not everything that counts can be counted. (Albert Einstein)
It is very difficult to establish what CAUSED outcomes – that is, to attribute
the change to your service/support. Be cautious in your conclusions.
Be careful not to spend more time in outcome measurement and its
complexities than in taking action to improve the lives of people

Relevant publications by research team
Campain, R.& Wilson, E. (under review) Life is what happens while you’re busy making plans: issues in
the development of outcome measurement for individualised plans for people with disability.
Quilliam, C. & Wilson, E. (2011). Literature Review - Outcomes measurement in disability services: a
review of policy contexts, measurement approaches and selected measurement tools.
Melbourne: Deakin University.
Hagiliassis, N., Nicola-Richmond, K., Mackay, A., & Wilson, E. (2011). Administration Manual -
Measuring Outcomes in Services and Supports (MOSS) tool: a means to assess the outcomes of
person-directed goals. Melbourne: Scope.
Nicola-Richmond, K., Wilson, E., Hagiliassis, N., & Mackay, A. (2011). Summary Sheet - Measuring
Outcomes in Services and Supports (MOSS) tool. Melbourne: Scope.
Quilliam, C., Wilson, E., Hagiliassis, N. & Nicola-Richmond, K. (2010). Measurement of Services and
Supports (MOSS): Project Overview and Findings. Melbourne: Deakin University.
Wilson, E. & Campain, R. (2011), Above and Beyond: Exploring outcomes and practices of Scope
Southern Region Early Childhood Intervention Service for children with disability, Melbourne:
Scope (Vic).
Wilson, E. & Campain, R. (2008). Problem solving, people skills, and personalised arrangements. Early
results of research into individualised approaches for young adults receiving ‘Futures’ funding at
Scope. C.Bigby & C. Fyffe (Eds.). Achieving their own lives. Proceeding from the Third Intellectual
Disability Policy Roundtable, La Trobe University, October
Wilson, E. (2006). Defining and measuring the outcomes of inclusive community for people with
disability, their families and the communities with whom they engage. C. Bigby; C. Fyffe & J.
Mansell (Eds). From Ideology to Reality: Current issues in implementation of intellectual disability
policy. Proceedings of the Roundtable on Intellectual Disability Policy. Bundoora: La Trobe
University
Wilson, E.; Jenkin, E. & Campain, R. (2011). Outcome Measurement of Community Based Mental
Health Services in Western Australia: Literature and Concept Summary. Melbourne: Inclusion
Matters
PLUS many conferences and workshops and seminars

References
ABS (2001) 4160.0 Measuring Wellbeing: Frameworks for Australian Social Statistics. Viewed at:
http://www.abs.gov.au/Ausstats/abs@.nsf/0/0120F9081822AAEBCA256B5F00804216?Open, on 31 October
2005.
Andreson E (2000), Tools of disability outcomes research: criteria for assessing the tools of disability outcomes
research, Archives of physical medicine and rehabilitation, Vol. 81, No. 12, pp. S1-S4.
Commonwealth of Australia (2011) . 2010-2020 National Disability Strategy. An initiative of the Council of Australian
Governments. Canberra.
http://www.fahcsia.gov.au/sites/default/files/documents/05_2012/national_disability_strategy_2010_2020.pdf
Department of Human Services (2012) Department of Human Services Standards self-assessment report and quality
improvement plan. http://www.dhs.vic.gov.au/about-the-department/documents-and-resources/reports-
publications/department-of-human-services-standards-self-assessment-report-and-quality-improvement-plan
Horsch, K. (2005). Indicators: Definition and use in a Results-Based Accountability System. Harvard Family Research
Project viewed at http://www.gse.harvard.edu/hfrp/pubs/onlinepubs/rrb/indicators.html on 31 Oct 2005.
King, S.; Rosenbaum, P. & King, G (1995). The Measure of Processes of Care: A means to assess family-centred
behaviours of health care providers, McMaster University, Neurodevelopmental Clinical Research, Hamilton,
Canada.
Schalock, R.L. (2004). The concept of quality of life: what we know and do not know, Journal of intellectual disability
research, 48, (3): 203-216.
Saunders, P., Naidoo, Y. & Griffiths, M. (2008). Left out and missing out: Disability and disadvantage. Social Policy
Research Centre, Sydney, Australia.
Saunders, P., Naidoo, Y. & Griffiths, M. (2007a). Deprivation & social exclusion in Australia. Social Policy Research Centre,
Sydney, Australia.
Saunders, P., Naidoo, Y. & Griffiths, M. (2007b). Towards new indicators of disadvantage: Deprivation and social
exclusion in Australia. Social Policy Research Centre, Sydney, Australia.
Triangle (n.d.) Outcomes Star. http://www.outcomesstar.org.uk/

Contact
Dr Erin Wilson
Senior Lecturer, School of Health & Social Development
Deakin University
erin.wilson@deakin.edu.au
Dr Nick Hagiliassis
Research Coordinator
Scope
nhagiliassis@scopevic.org.au

Dr Erin Wilson - School of Health and Social Development, Deakin University

Recommended

Recommended

More Related Content

Viewers also liked

Viewers also liked (17)

More from field - furthering inclusive learning and development

More from field - furthering inclusive learning and development (20)

Recently uploaded

Recently uploaded (20)

Dr Erin Wilson - School of Health and Social Development, Deakin University