1. A Workshop on the Basics of Systematic Review & Meta-Analysis Philip C. Abrami, Robert M. Bernard C. Anne Wade, Evgueni Borokhovski, Rana Tamim, Gretchen Lowerison & Mike Surkes Centre for the Study of Learning and Performance and CanKnow Concordia University

4. 02/25/11

5. Purpose: Explaining Variability in Effect Size 02/25/11 Effect Sizes Study Features Shared Variability Unique Variability Unique Variability Prediction

7. 02/25/11

8. 02/25/11 1. Determine the research question The “big question” that guides the research. It usually involves asking about the difference between two conditions (i.e., usually treatment and control) or the relationship between two measures. 10 Steps in a Meta-Analysis

10. Example: Critical Thinking 02/25/11 Research Question: What instructional interventions, to what extent, and under what particular circumstances, impact on the development and effective use of learner’s critical thinking skills and dispositions?

11. 02/25/11 2. Develop terms and definitions related to the question This helps refine the research question and inform the search strategies. 10 Steps in a Meta-Analysis

12. 02/25/11 3. Develop a search strategy for the identification of relevant studies This involves the planning/implementation of search and retrieval for primary studies (e.g., electronic databases, branching). 10 Steps in a Meta-Analysis

14. 02/25/11 Preliminary Searches Reference Sources: Purpose : To obtain definitions for the terms; creativity, critical thinking, decision making, divergent thinking, intelligence; problem solving, reasoning, thinking. Sources: Bailin, S. (1998). Critical Thinking: Philosophical Issues . [CD-ROM] Education: The Complete Encyclopedia. Elsevier Science, Ltd. Barrow, R., & Milburn, G. (1990). A critical dictionary of educational concepts: An appraisal of selected ideas and issues in educational theory and practice (2 nd ed.) . Hertfordshire, UK: Harvester Wheatsheaf Colman (2001). Dictionary of Psychology (complete reference to be obtained) Corsini, R. J. (1999). The dictionary of psychology . Philadelphia, PA: Brunner/Mazel Dejnoka, E. L., & Kapel, D. E. (1991). American educator’s encyclopedia . Westport, CT: Greenwood Press. …… (see handout)

21. Documenting Your Searches 02/25/11 Example from our Codebook: ERIC (Date: September 21, 2003; AW) Purpose : To retrieve the first set of abstracts to be reviewed by team according to the current inclusive/exclusion criteria. Result : Hit rate of 514/1520 Source code: ERIC1 Searches and records below from: The ERIC Database (1966-2003, June) #5 #3 and #4 (1520 records) #4 DTC = 142 or DTC = 143 or control group* (322893 records) #3 #1 or #2 (7718 records) #2 critical thinking in DE,ID (7562 records) #1 thinking skills in DE and critical thinking (1269 records)

25. 02/25/11 10 Steps in a Meta-Analysis 4. Establish criteria for inclusion and exclusion of studies These are the criteria that guide the search for literature and ultimately determine what studies are in and out of the review.

27. 02/25/11 10 Steps in a Meta-Analysis 5. Select studies based on abstract review This is the initial decision as to what studies will be retrieved as full-text documents.

28. 02/25/11 10 Steps in a Meta-Analysis 6. Select studies based on full-text review This is the second decision as to what studies will be included in the review.

29. 02/25/11 10 Steps in a Meta-Analysis 7. Extract effect sizes Effect sizes extraction involves converting descriptive or other statistical information contained in studies into a standard metric by which studies can be compared.

33. 02/25/11

34. 02/25/11

35. Example of ES Extraction with Descriptive Statistics 02/25/11 Study reports: Treatment mean = 42.8 Control Mean = 32.5 Treatment SD = 8.6 Control SD = 7.4 n = 26 n = 31 Procedure: Calculate SD pooled Calculate d and g

37. Examples of Alternative Methods of ES Extraction 02/25/11 • Study Reports: t (63) = 2.56, p < .05 • Study Reports: F (1, 63) = 2.56, p < .05 Convert F to t and apply the above equation:

38. Zero Effect Size 02/25/11 ES = 0.00 Control Condition Treatment Condition Overlapping Distributions

39. Moderate Effect Size 02/25/11 Control Condition Treatment Condition ES = 0.40

40. Large Effect Size 02/25/11 Control Condition Treatment Condition ES = 0.85

41. Mean and Variability 02/25/11 Variability ES+ Note: Results from Bernard, Abrami, Lou, et al. (2004) RER

42. 02/25/11 10 Steps in a Meta-Analysis 8. Develop a codebook Study feature coding involves describing the relevant characteristics for each study (e.g., research methodology, publication source).The codebook details the study feature categories and their levels.

45. 10 Steps in a Meta-Analysis 02/25/11 9. Code studies for study features Coding study features is perhaps the most time-consuming and onerous aspect of conducting a meta-analysis. However, it is arguably the most important step because it provides the possibility for explaining variability in effect sizes.

46. 02/25/11 10 Steps in a Meta-Analysis 10: Analysis and interpretation Analysis involves invoking a range of standard statistical tests to examine average effect sizes, variability and the relationship between study features and effect size. Interpretation is drawing conclusion from these analyses.

49. 02/25/11

50. 02/25/11 Statistics in Comprehensive Meta-Analysis™ Comprehensive Meta-Analysis 2.0 is a trademark of BioStat® Interpretation: Moderate ES for all outcomes ( g+ = 0.34) in favor of the intervention condition. Homogeneity of ES is violated. Q-value is significant (i.e., there is too much variability for g+ to represent a true average in the population).

51. Examining the Study Feature “Type of Research Design” 02/25/11 g + = +0.34 Overall Effect Pre-Post Designs Post-Only Designs Quasi-Exp. Designs

52. Tests of Levels of “Type of Research Design” 02/25/11 Interpretation: Small to Moderate ESs for all categories in favor of the intervention condition. Homogeneity of ES is violated. Q-value is significant for all categories (i.e., type of research design does not explain enough variability to reach homogeneity.