Presentation by Rob Briner. CEBMa's Annual Meeting 2018, Chicago

1. 1
Using CATs and REAs
to inform decision-making
CEBMa Annual Meeting: How to Teach
and apply Evidence-Based Practice
theWit Hotel Chicago 9 August 2018
Rob B Briner
www.cebma.org

2. 2
What is it?
 The conscientious (effort), explicit
(clarity) and judicious (critical of quality)
use of the best available evidence from
multiple sources to increase the likelihood of
a favourable outcome
– It’s about the process
– It’s not about certainties (this will work)
– It’s is about probabilities and likelihoods
– It is about reducing uncertainty (given our context
this is more likely to lead to the outcome we want
than doing something else or doing nothing)
2

3. 3
3

4. 4
4

5. 5
5

6. 6
What are CATs and REAs?
 Critically Appraised Topics
 Rapid Evidence Assessment
 They are shorter versions of systematic
literature reviews of the literature
 They are the “scientific literature” part of
EBMgt
6

7. 7
Possible purposes and properties of
literature reviews
 Critical?
 Comprehensive?
 Detailed?
 Exhaustive?
 Objective?
 Focused?
 Descriptive?
 Practically relevant?
 Analytic?
 Theoretical?
 Gap-identifying?
 Summarizing?
 Selective?
 Interesting?
 Thematic?
 Systematic?
7

8. 8
What types of literature review are
conducted in management?
 Literature reviews motivating empirical
studies
 Formal full-length literature reviews by
academics
 Meta analyses
 Reviews or overviews in textbooks
 Reviews in popular management books
 Reviews in management usually narrative not
systematic 8

9. 9
Nature of claims made by most of these
reviews
 For example:
– Previous research has shown that team building
improves performance
– It has been demonstrated that management
development is effective
– Many studies have shown that employee
engagement increases performance
– There is much evidence that job stress causes ill
health
9

10. 10
Nature of claims made by most of these
reviews
 BUT!
– Did all previous research show this?
– What proportion of previous research?
– How many studies?
– How strongly or clearly or consistently was this
shown?
– Were the study designs such that the conclusions
reached could be justified?
– What did the authors do to avoid the biases of
pre-existing beliefs?
10

11. 11
Disadvantages of traditional
management literature reviews
 Do not usually focus on a specific (or practice-relevant) question
 Have diverse aims and purposes
 Adopt a wide variety of approaches and structure
 Do not use particular methods or do not explicitly state the methods
used to conduct the review
 Are more prone to bias because, for example, authors can select
studies that support their views and ignore those that do not
 Are less likely to identify the best available evidence
 Do not aim to be comprehensive by including all relevant and available
research
 Are not replicable nor easy to update
 Do not tell us what we know and do not know and the bases for that
ignorance/knowledge
 Cannot ‘stand on the shoulders of giants’ (and all research findings)
11

12. 12
What are systematic reviews?
 They are research on existing research
 Precise question (like a research question)
 Explicit methodology
 Replicable
 Make it clear:
– What is known
– What is not known
– And the basis for those claims
12

13. 13
What are they?
 “systematic reviews never provide ‘answers’.
What they do is report as accurately as
possible what is known and not known about
the questions addressed in the review”
(Briner, Denyer, & Rousseau, 2009, p. 27).
13

14. 14
Core principles of REAs, CATs and SRs
 Systematic/organized: Systematic reviews are
conducted according to a system or method which is
designed in relation to and specifically to address the
question the review is setting out to answer.
 Transparent/explicit: The method used in the review is
explicitly stated.
 Replicable/updatable: As with many forms of primary
research, the method and the way it is reported should be
sufficiently detailed and clear such that other researchers
can repeat the review, repeat it with modifications or
update it.
 Synthesize/summarize: Systematic reviews pull
together in a structured and organized way the results of
the review in order to summarize the evidence relating to
the review question.
14

15. 15
CEBMa guidelines
15

16. 16
Structure of a CAT?
 Section 1. Background: What is the context of the CAT question?
 Section 2. What does the CAT answer?
 Section 3. Defining inclusion criteria: Which studies will be taken into
account?
 Section 4. Search strategy: How should the studies be sought?
 Section 5. Study selection: How should you select the studies?
 Section 6. Data extraction: What information should be extracted?
 Section 7. Critical appraisal: How should the quality of the studies be
judged?
 Section 8. Results: What did you find?
 Section 9. Conclusion
 Section 10. Limitations
 Section 11. Implications and recommendations
16

17. 17
17

18. 18
Review questions
 What is known in the scientific literature about the impact of (1)
goal setting and (2) performance appraisal on workplace
performance?
 What is meant by (1) goal setting and (2) performance
appraisal? (What are they?)
 What are the assumed causal mechanisms? (How are they
supposed to work?)
 What is the effect of (1) goal setting and (2) performance
appraisal on workplace performance?
 What is known about possible moderators and/or mediators that
affect the relationship between (1) goal setting and (2)
performance appraisal and workplace performance?
 What is known about the reliability and validity of performance
appraisal?
18

19. 19
Example CAT topics from Masters in HRM
 What is known in the scientific literature about
– the effects of flexible working interventions on: (a) Work-nonwork
conflict? And, (b) Psychological well-being?
– interventions to increase innovative behaviour at work?
– (a) The validity and reliability of measures of unconscious bias? and,
(b) The effects of unconscious bias training on unconscious bias?
– the effects of mindfulness training on the well-being and performance
of employees?
– the effects of interventions aimed at improving safety culture?
– the effects of interventions aimed at reducing sexual harassment?
– the effects of training interventions designed to improve multicultural
working in teams
– interventions to increase the percentage of women in senior
management positions in large organizations?
– interventions and techniques which aim to improve the performance of
employees identified as poor performers?
19

20. 20
Using CATs and REAs to inform decision-
making
20

21. 21
Example of employee engagement (EE)
 Suppose you or your organization believe
that low EE is somehow a problem in your
organization
 And because it is you should measure EE and
attempt to increase ‘low’ scores somehow
 How would you (a) approach this problem
and (b) identify a solution from an
evidence-based practice approach?
21

22. 22
22
Element 1: Practitioners’ professional
expertise
 Identifying the problem
– Have I/we seen EE problems before? What
happened?
– Based on our experience, is the level of EE a
problem?
– What do I/we believe about causes and
consequences of low EE?
 Identifying solution (only if EE is a problem)
– Have I/we seen EE interventions before? What
happened?
– What do I/we believe about EE interventions?
– Based on our experience, is the level of EE here a
problem? What are costs & benefits of intervening?
 How relevant and applicable and trustworthy is
my/our expertise?

23. 23
23
Element 2: Organizational data
 Identifying the problem
– What actually is the EE level?
– Are there patterns or trends in EE?
– Do data show how EE is a problem?
– Do data show that low EE is causing problems?
 Identifying solution (only if EE is a problem)
– What attempts to enhance EE are currently in place
and are they working?
– What else is happening that might be affecting EE?
– Are there relationships between EE and other data?
Employee type? Shift? Department?
 How relevant and applicable and trustworthy are
our organizational data?

24. 24
24
Element 3: Scientific literature
 Identifying the problem
– What are the average rates of EE in my sector and
location – is the level here ‘low’?
– What does systematically reviewed research evidence
suggest are the problems with low EE?
 Identifying solution (only if EE is a problem)
– What does research evidence from systematic
reviews suggest are major causes of low EE?
– What does research evidence from systematic
reviews suggest as effective interventions to increase
EE?
 How relevant and applicable and trustworthy are the
scientific findings?

25. 25
25
Element 4: Stakeholders values and
concerns
 Identifying the problem
– How do employees feel about and view the EE
‘problem’?
– Do they see negative consequences?
– What do managers think about the problem?
– Do customers or clients or service users have a view?
 Identifying solution (only if EE is a problem)
– How do employees feel about and view the solutions?
– What do managers think about the solutions?
– What alternative explanations and proposed solutions
do others have?
 How relevant and applicable and trustworthy are
stakeholder concerns evidence?

26. 26
How would you use the scientific
evidence to help make a decision?
26

27. 27
The value of aggregation: Scientific
evidence in context
 Is there in principle likely to be a practical problem?
 Is the proposed solution in principle likely to be effective
 Scientific evidence is not just about findings – it elaborates
understanding of the problem and potential solution
 Helps tell you extent to which scientific evidence is
relevant to your context
 Helps to redefine nature of problem and solution – we
thought it was X but more likely is Y
 May give insight into causal pathways
 Indications about how important problem/solution is
compared to others (X seems to influence Y but effect size
much smaller than A, B, and C)
27

28. 28
Questions? Comments? Thoughts?
r.briner@qmul.ac.uk
Rob B Briner
www.cebma.org