3. Improving Futures
Key Indicators
• Launched by the Big Lottery Fund in March 2011
• Total £26 million grant funding distributed between 26 pilot
projects across the UK
• Aiming to test different approaches to improve outcomes
for children in families with multiple and complex needs
• Requirements:
– led by VCOs partnership with statutory services
– include mechanisms to engage 'hardest to reach' families; and.
– oldest child 5-10 years at entry stage
4. Aims of the Evaluation
Measuring Impact
Evaluation aims
1. To measure the effectiveness, impact and outcomes of the
26 projects; and the programme as a whole
2. To support the projects with measuring progress
3. To capture and share learning across the programme, and
to disseminate to policymakers & practitioners across the UK
Scope
• Consortium led by Ecorys, with Ipsos MORI, Family Lives and
University of Nottingham
• Five year evaluation (Oct 2011 - Dec 2016)
• Evaluation and learning
5. Evaluation Framework
Programme Effectiveness
Programme Impacts
Programme-level
Effectiveness of programme design,
development and implementation
Net improvements to children‟s life chances
Net risk reduction for families with complex needs
Net attributable savings
Project-level
Effectiveness of governance;
partnerships; strategy and planning;
user involvement and delivery
Quality of provision
Sustainability
Strategic Benefits
Strategic influence over UK policy
Leverage over external resources
Synergy with other programmes
Knowledge transfer
Replication of effective practice
Programme Outcomes
Children
Adults
Families
Health and
wellbeing
Parenting skills
and confidence
Emotional and
behavioural
development
Health and
wellbeing
Family
functioning and
relationships
Educational
outcomes
Safeguarding and
social care
Crime and ASB
Educational
outcomes
Safeguarding and
social care
Crime and ASB
Financial
wellbeing and
security
Secure and safe
environment
Social networks
and belonging
6. Methodology
Measuring Impact
Programme-Level Evaluation
•
Literature review
•
Project-level evaluation
Monitoring data collection (IFMIS):
secure online system (practitioner-led)
•
Co-production of local evaluation
plans: support with identifying project
outcome measures & data sources
•
Bi-annual monitoring against local
grant milestones, and qualitative
feedback on lessons learned
•
Longitudinal survey of families:
baseline; +12 mths, and +24 mths
•
Stakeholder telephone survey: key
local stakeholders (2013 & 15)
•
Support for self-evaluation: advice
on methods, tools and training
•
Case study research: mix of
longitudinal (n=6) & snapshot (n=14)
•
•
Family Advisory Panel
Enhanced evaluation activities: CBA
case studies; impact case studies;
participatory research case studies
7. IFMIS: Overview
Measuring Impact
Key considerations
• Need to gather comparable data: measuring
changes to families‟ status over time
• 26 local projects, with different local data systems
Format
Secure online database
Data entry by project workers, based upon data
recorded within service plan (or equivalent)
Three mandatory data entry points: entry, exit, +
6mths post-exit (with optional interim measure)
8. Key Indicators
• Child, adult and family indicators, based on a review of the
research literature, plus socio-demographic data
• Problems and strengths; hard and softer measures
• Three indicator types : dynamic (subject to change), eventbased (time-specific), and status (interpretive only)
• Record type and amount (hours) of support received
11. Measuring Impact and Outcomes
Measuring Impact
1. Desk research to develop key indicators (risk factors & strengths)
2. Embed within monitoring framework (IFMIS)
3. Measure short-term change over time (monitoring data)
4. Measure longer-term outcomes, up to +2yrs (survey)
5. Test predictive „strength‟ of indicators (compare survey & MI)
6. Scale-up to estimate programme-wide effects
12. Counterfactual
Measuring Impact
•
Need to establish „additionality‟ of the programme
•
Counterfactual analysis: explore alternative scenario
(no intervention). Two approaches:
1. Self-reporting of impacts: all projects (exit
questionnaire and via Ipsos MORI survey)
2. Quasi-experimental design: 4-5 impact case
study projects: compare outcomes between
participants and matched comparison group
(“difference-in-differences”)
13. Evaluation Activities Completed to Date
1. IFMIS system live and fully embedded
2. Bespoke evaluation plans in place for all 26 projects
3. Baseline survey of families underway
4. First local stakeholder survey underway
5. Six „trailblazer‟ case study visits completed (England,
Wales, and Scotland)
6. Participatory action research training delivered
14. Learning and dissemination strand
Measuring Impact
Policy Makers,
Practitioners, Wider
Stakeholders
Policy
Briefings
Good Practice
Publications
Project
Exchanges
Policy
Workshops
Webinars
Improving
Shared Learning Environment
Futures Wider Stakeholders
Policy Makers, Practitioners,
Projects
Face to Face Learning
Events
Electronic
Newsletters
Press
Releases
15. Learning activities completed to date
Measuring Impact
Internal
1.
2.
3.
4.
National launch event for the programme
Secure online portal for projects
Project networking events and webinars
Newsletter
External
1. Launch of evaluation web-site:
http://www.improvingfutures.org/
2. Social media activities
3. Policy round table session
17. Part 2:
Key Findings from
Year 1 of the Evaluation
James Ronicle, Evaluation Project
Manager
18. Introduction
-
Progress in Establishing Projects
-
Supporting Families
-
Who are the Improving Futures Families?
-
Early Outcomes for Families
-
Conclusions and Next Steps
-
Implications and Lessons Learnt for Future BIG
Programmes
19. Progress in Establishing Projects
• Focus to date has been on building local partnerships and delivery teams.
• Projects still in early stages of delivery & models are evolving
• Number of families supported by projects are lower than anticipated.
However, referral numbers picked-up considerably during 2013.
• Projects face more challenging financial climate than when launched.
• Projects faced barriers to partnership working due to heightened sense of
competition & widespread restructuring
• Despite this, projects building strong local partnerships:
- Engaging public services
- Collaborations between VCSs
- Engaging „micro‟ VCSs
• Consequently, projects sharing best practice & early signs becoming
embedded within local structures for children & families.
20. Supporting Families:
Supporting Families:
Examples of Improving Futures Projects
Examples of Projects
The Neighbourhood Alliance,
Sunderland
•
•
Led by Foundation of Light, Sunderland Football Club
Developed „Neighbourhood Menu‟ of 45 „micro- VCSs‟ to
support families
Family Pathways, Lewisham
•
•
•
Focuses on children demonstrating destructive or anti-social
behaviour
Runs „Roots of Empathy‟; classroom-based programme
where children spend time with babies to strengthen
emotional competence
Delivers training for
practitioners on promoting
attachment between
parents & children
BIG Manchester
Tyne Gateway, N & S Tyneside
•
•
•
Created co-located team to tackle „toxic trio‟ of substance
misuse, mental health & domestic violence
Provides three areas of support:
•
Holistic family support through Key Worker
•
Peer and community engagement
•
Spot purchasing of specialist support
Recruits Family Entrepreneurs as „barefoot professionals‟:
•
Outreach
•
Support in assessing family needs
•
Support in delivering services
•
Direct support to families
21. Supporting Families:
Identification and Engagement
• Primary schools have provided a hub for many
projects (59% of families engaged through schools)
• Challenges around reaching out to ‘harder to
engage’ families who do not traditionally participate
• Previous negative experiences of contact with
services having an impact on willingness to engage
• Some challenges around 5-10 age criteria
• Greater number of families with complex needs
than anticipated, in some projects
22. Supporting Families:
Packages of Support
• Replicating many aspects of good practice :
•
•
•
„Key Worker‟ model
Holistic support packages
Evidence-based parenting programmes:
• Incredible Years, Triple P, Roots of Empathy
• Innovative finance mechanisms:
•
•
Family budgets
Spot purchasing
• Peer support
• However, not all projects providing ‘holistic’ support
• Key success factors:
•
•
•
„Out of hours‟ provision
Frequent Key Worker contact
Hands-on approach with referrals to other services
23. Who are the Improving Futures Families?
Measuring Impact
Demographics
• Free School Meals significantly above national
averages (77%)
• Minority ethnic background – above UK averages
(at 31%). Variation from previous family programmes
(FIPs: 88% white, and FPs: 77% White-British)
• High proportion of lone parent families –
approaching two thirds, at entry stage (62%).
• Predominately female adult profile (75%)
24. Who Are the Improving Futures Families?
Family Risk Factors
Measuring Impact
• ‘Upper-middle’ in continuum of need. More prevalent than general population;
less prevalent than other intensive intervention programmes.
Figure 1: Family risk factors affecting 10% or more of the cohort
Base: 891 families
25. Who Are the Improving Futures Families?
Measuring Impact
Adult indicators
Risks (-)
Strengths (+)
•
•
Parenting difficulties the most
prevalent issue: parenting anxiety or
frustration (56%), problems with
discipline / boundaries (49%)
Over half of adults within the cohort
(51%) demonstrating awareness of
how to keep children safe from
harm, and around one third setting
appropriate boundaries (34%)
•
Well over one third with suspected
or reported stress or anxiety (41%
- twice the population average)
•
Far lower prevalence of adult
ASB or crime indicators than
some other programmes (2.1%
ASB, compared 79% for FIPs).
Fair levels of face-to-face contact
with school staff and positive
relationships, ranging from under
half for all adults (45%) to just over
half for lone parents (52%)
•
26. Who Are the Improving Futures Families?
Measuring Impact
Child indicators
Risks (-)
Strengths (+)
•
Child behavioural problems by far •
the most prevalent risk, when lowlevel (30%) and persistent disruptive
behaviour (16%) are combined
•
One in five children with suspected
or reported stress or anxiety
(22%); mirroring similar problems in
adults within the cohort, and one in
ten under-achieving at school
•
Lower propensity for project workers
to report risk factors for children;
could indicate recording difficulties
•
Most children adopt some aspects
of a healthy lifestyle: well over half
(61.5%) attend regular medical
appointments; a fair proportion
participate in play (37%), and
exercise or sport activities (27%)
Over one third of children have
supportive peer friendships at
school (43%), whilst just over one
quarter have regular contact with
friends outside school (27%).
27. Who Are the Improving Futures Families?
Measuring Impact
Towards some typologies…
• Factor analysis of IFMIS data (in SPSS) – clusters
of variables that are strongly correlated with each
other but relatively uncorrelated with variables in
other clusters
• Iterative approach to find the most robust and
realistic groupings
• Eight typologies identified on this basis...
28. Who Are the Improving Futures Families?
Measuring Impact
Towards some typologies (2)
• Typology 1 - Families with strong structures and behaviours
• Typology 2 – Low skilled families with financial and housing difficulties
• Typology 3 – Families with significant discipline and behavioural issues
• Typology 4 – Transient families with domestic abuse, mental health and
substance misuse
• Typology 5 – Families engaged in their community
• Typology 6 – Families known to the police for minor disorder
• Typology 7 – Socially excluded families with health problems
• Typology 8 – Families known to the police for more serious incidents
29. Early Outcomes for Families
1. Projects able to achieve „quick wins‟ for families:
1. Resolving financial problems
2. Improving housing circumstances
2. Parents reporting improved well-being:
1. Confidence & self-esteem
2. Reduced Anxiety
3. Parenting capacity
3. Improved behaviour and attendance at school
30. Conclusions and Next Steps
Evaluation Conclusions
•Improving Futures is becoming integrated into wider local support
infrastructure for children and families
•The programme is strengthening links between statutory and VCS
organisations
•Projects showcased diverse range of practices for working with
families at different levels of need. Some are replicating good practice,
others developing good practice
•Too early to offer robust conclusions about impact of programme, but
positive early signs that projects making different to families‟ lives.
Areas for Development
•Reaching families not in contact with mainstream services
•Taking stock of ‘whole family’ approaches
31. Implications / Lessons Learned
for Future Programmes
Evaluation design
• Need to strike a balance between a) freedom for grant-holders to
innovate, and b) setting parameters for measuring impact
Programme design
• Value of statutory (DCS) endorsement of VCS bids
• Development phase critical for building multi-agency partnerships
(…but make next stages conditional on early success?)
Key areas of added value / potential legacy…
•
•
•
•
Understanding dynamics of change for families over time
What works with the 5-10 age group, and VCS-school links
New forms of service commissioning (esp. neighbourhood level)
Models of engagement and support for BME families
