Public involvement in the systematic review process in health and social care: A narrative review of case examples.

1. Public involvement in the systematic review process in
health and social care: A narrative review of case
examples
9th HTAi Annual Meeting, Bilbao, Spain, 25-27th June 2012
Jonathan Boote (1), Wendy Baird (1) and Anthea Sutton (2)
(1)
NIHR Research Design Service for Yorkshire and the Humber, (2) The University of
Sheffield

2. Background to the review
• Perhaps the most important type of health research that the public can
become involved in is the systematic review
– Systematic review has high status in evidence hierarchies
• Impetus for involving the public in the systematic review process has come
primarily from the Cochrane Collaboration (Bastian, 1994; Horey, 2010)
– The public have been part of the Collaboration since its inception
– Public representation on the Cochrane steering group
– Cochrane Consumer Network
• 2 surveys of the extent to which Cochrane review groups involve the public
have been published : Kelson, 1999, Wale et al, 2010
• 4 Cochrane review groups have published accounts of public involvement
within their respective groups:
– Musculoskeletal (Shea et al, 2005), Pregnancy and Childbirth (Sakala et
al, 2001), Breast cancer (Ghersi, 2002), Haematological malignancies
(Skoetz et al, 2005)

3. Background to the review
• Detailed published accounts of public involvement within individual
systematic reviews remain scarce
• Purpose of the review was to identify and review case examples of
public involvement in the design and conduct of individual systematic
reviews, to:
– examine the methods, levels and stages of involvement reported
– synthesise the identified contributions of involving the public
– discuss the tensions, facilitating strategies and recommendations
for good practice that have been identified
• Excluded from the review were:
– Viewpoint or discussion papers on public involvement in
systematic reviews
– Surveys of public involvement in systematic reviews.
– Documents with insufficient detail of the contribution(s) of the
public to the systematic review.

4. Searching for evidence
• A literature search was undertaken in January 2011 of the following
databases, with the aim of identifying peer-reviewed journal articles
and reports published in English on public involvement in the
systematic review process:
– PsycINFO, MEDLINE, and MEDLINE In-Process, CINAHL,
EMBASE, Cochrane Methodology Register, HMIC, invoNET.
• Search terms included ‘consumer’, ‘user’, ‘participant’, ‘involvement’,
‘inclusion’, ‘review’, ‘systematic review’ and ‘research’.
• A further search of a comprehensive bibliography of references on
public involvement in health and social care research (Boote et al,
2012) was undertaken, to identify relevant reports of systematic
reviews that involved the public

5. Identifying evidence
• A total of 744 documents were identified in the search of databases,
discounting duplicates
• abstracts were scrutinised for possible inclusion in the review by the
lead researcher, and these inclusions were checked for accuracy by a
second researcher
• Of these 744 documents, three papers were identified on public
involvement in the systematic review process on the basis of paper
title and abstract (where available)
• The search of the bibliography identified a further four further relevant
sources.
• In total, seven case examples were found.

6. Included case examples (1)
• Braye S, Preston-Shoot M. Emerging out of the shadows? Service user and
carer involvement in systematic reviews. Evidence and Policy: A Journal
of Research, Debate and Practice 2005;1(2):173–94.
• Smith E, Donovan S, Beresford P, Manthorpe J, Brearley S, Sitzia J, et
al.Getting ready for user involvement in a systematic review. Health
Expectations 2009;12:197–208.
• Serrano-Aguilar P, Trujillo-Martín MM, Ramos-Go˜ni JM, Mahtani-Chugan V,
Perestelo-Pérez L, Posada-de la Paz M. Patient involvement in health
research: a contribution to a systematic review on the effectiveness of
treatments for degenerative ataxias. Social Science and Medicine
2009;69(6):920–5.
• Brett J, Staniszewska S, Mockford C, Seers K, Herron-Marx S, Bayliss H.
The PIRICOM Study: a systematic review of the conceptualisation,
measurement, impact and outcomes of patients and public involvement in
health and social care research. UK Clinical Research Collaboration;
2010, http://www.ukcrc.org/index.aspx?o=3233

7. Included case examples (2)
• Nilsen ES, Myrhaug HT, Johansen M, Oliver S, Oxman AD. Methods
of consumer involvement in developing healthcare policy and
research, clinical practice guidelines and patient information material.
Cochrane Database of Systematic Reviews
2006;3(September),ncbi-p:1469493X.
• Carr S, Fleischmann P. Systematic review of consumers’ perspective
on electro-convulsive therapy. In: Carr S, Coren E, editors. Collection
of examples of service user and carer participation in systematic
reviews. London: SCIE;
2007.http://www.scie.org.uk/publications/researchresources/rr02.pdf
• Rees R, Oliver S. An example from sexual health promotion. In: Carr
S, Coren E, editors. Collection of examples of service user and
carer participation in systematic reviews. London: SCIE;
2007.http://www.scie.org.uk/publications/researchresources/rr02.pdf

8. Summary of included papers (1)
Paper and Topic area Public Stage, method and type of
country involvement
Braye and teaching, learning Wide range of different Consultation, advisory group,
Preston-Shoot, and assessment of types of public define the scope and interpret
2005, UK law in social work findings
education
Smith et al, User involvement Members of 24 Collaboration (user-
2009; UK in nursing, midwifery national service researcher) and consultation
and health visiting user organisations (advisory group)
research
Serrano- Degenerative 53 patients with Consultation - scoping the
Aguilar et al, ataxias degenerative review, Choice of outcomes,
2009 ataxias Delphi method
Carr and Patients’ 2 service user User-led, All stages of the
Fleischman et perspectives on researchers with direct research. Use of user-
al, 2007 Electro-convulsive experience of ECT and researchers and advisory
therapy representatives of user group
and voluntary groups
with a stake in ECT
research

9. Summary of included papers (2)
Paper and Topic area Public Stage, method and type
country of involvement
Rees and HIV health promotion for Representatives of a Consultation – define the
Oliver, 2007. men who have sex with number of health scope of the review,
UK men promotion charities e.g. agree outcomes and
Terrence Higgins Trust comment on findings.
Advisory group
Brett et al, The conceptualisation, 3 lay people on advisory Collaboration –
2007. UK measurement, group. Members of membership of advisory
impact and outcomes of UNTRAP and Diabetes group; consultation via a
public involvement in User Network Research workshop to discuss
health research Group findings
Nilsen et al, Methods of consumer A panel consisting Consultation through
2006 involvement in developing of members of the email discussion list.
healthcare policy and Cochrane Consumer Consulted on the review
research, clinical practice Network. protocol, literature search,
guidelines and patient and the initial draft of the
information material review

10. Reported contributions of the public to
systematic reviews
1. Refining the scope of the review
– In the review of HIV health promotion for MSM, the public identified HIV+ and
young MSM as population subgroups on which the review should focus
1. Suggesting and locating relevant literature
– In the review of patients’ experiences of ECT, a user group located an
unpublished study of ECT for the benefit of the review
1. Appraising the literature
– User researchers were involved in the review of patients’ experiences of ECT
and appraised the literature
1. Interpreting the findings
– Two reviews reported the use of consultation workshops where preliminary
findings were discussed with the public
1. Writing up the review
– Three of the reviews reported the public being involved in the writing up stage of
the review process: either as first author, co-author, or as the author of the
review’s foreword

11. Tensions identified when involving the
public in the review process
• Time pressures
– It can take time to build up trust between researchers and the public in the
review process
• Resourcing problems
– Lack of funding at the review development (pre-grant) stages to support
involvement
• Continuity issues
– It can be difficult for the public to attend all review meetings
• Concerns about group dynamics
– The user-led review of experiences of ECT reported tensions regarding the
amount of power the user-researchers had in the review process
• Research Ethics Committee involvement
– One review reported that they had to obtain approval from a REC to involve the
public in the review process
• Tensions associated with the representativeness of the
members of the public involved
– Some reviews mentioned that those who get involved in reviews will not be
representative of the wider population

12. Good practice recommendations for involving
the public in systematic reviews
• Funding and payment
– The public should be paid for the time they spend contributing to reviews
and that this should be budgeted for
• Identifying a lead for public involvement
– This person should be the contact point for the public and this role should
be costed into the review budget
• Training, briefing and information provision
– Provision of training, briefing notes and background information (such as a
glossary) has been recommended. The extent of this will vary depending
on the level of involvement.
• Structured methods of involvement
– Two reviews used structured processes (Delphi and voting at meetings) to
capture the views of the public during the review process. This prevents
the review process being dominated by particular, more vocal individuals.

13. Conclusions and implications
• The review found examples of the public being involved in the systematic
review process at the 3 main levels of involvement (consultation,
collaboration and user control) and using a variety of methods, including the
Delphi process, advisory group meeting and consultation workshops
• It was found that the public were involved at all the key stages of the review
process, but no reports were found of the public being involved in meta-
analysis of quantitative data
Implications:
• Commissioners of systematic reviews need to recognise that public
involvement in the review process means that reviews take longer to deliver
whilst being more expensive
• Financial support is needed to resource public involvement at the design
(pre-grant) stage of systematic reviews
• Abstracts of systematic reviews should describe the extent of public
involvement (if any) in the review process to facilitate more accurate future
reviews of the evidence base

14. Further reading
• Boote J, Baird W, Sutton A (2011), Public involvement in the systematic
review process in health and social care: a narrative review of case
examples. Health Policy, 102, 105– 116.
• Kreis J, Puhan M,Schunemann H, Dickersin K (2012), Consumer
involvement in systematic reviews of comparative effectiveness research.
Health Expectations, DOI: 10.1111/j.1369-7625.2011.00722.x
– Explores current approaches to involving consumers in US-based and key
international organizations and groups conducting or commissioning
systematic reviews
• Vale C, Tierney J, Spera N, Whelan A, Nightingale A, Hanley B (2012),
Evaluation of patient involvement in a systematic review and meta-
analysis of individual patient data in cervical cancer treatment.
Systematic Reviews 2012, 1:23 doi:10.1186/2046-4053-1-23
– Key example of involvement of the public in a meta-analysis along with an
evaluation of the process

15. Contact details:
Dr Jonathan Boote
Research Fellow and PPI Lead
NIHR Research Design Service for Yorkshire and the Humber
Email: j.boote@sheffield.ac.uk
Tel: 0114 222 0892
RDS website: http://www.rds-yh.nihr.ac.uk/