This is the handout version of a lecture I give to medical residents and fellows on the basics of clinical research designs and the inherent issues that go along with each one. I give this lecture as part of a multi-module lecture series on research design and statistical analysis.
1. Comparing
Research
Designs
Patrick Barlow
Statistical and Research Design Consultant, Graduate School of
Medicine, UTK
PhD Student in Evaluation, Statistics, and Measurement, UTK
2. On the Agenda
Common problems in research design
Confounding
Bias
Reliability
Validity
Observational Research Designs
Cross-Sectional study
Case-control study
Cohort study
Experimental Research Designs
Randomized Control Trial
4. Confounding
A confounder is a variable that is causally
associated with the outcome (DV) and may or may
not be causally associated with the exposure (IV)
Causes spurious conclusions & inferences to be
made about a set of variables
Reduced through
Randomization
Matching
Statistically controlling (covariates)
5. Confounding
Obesity
PMH Use
?
Colorectal
Adenomas
6. Bias in Research
The result of systematic error in the design or
conduct of a study
Can artificially “trend” results
Toward the Null hypothesis
Toward the Alternative hypothesis
A major problem to consider when planning any
study
7. Common Biases
Selection bias: one relevant group in the population (e.g.
cases positive for predictor variable) has a higher
probability of being included in the sample
Ascertainment: bias in asking questions or offering tests
of one group over another
Information: bias from erroneously classifying people in
exposure/outcome categories
Adjudication: bias in determining if the treatment was
helpful due to partial or inadequate blinding
Recall/Response: bias associated with inaccurate recall
of exposure or representation of true exposure (self-
report)
Experimenter/Interviewer bias: Differential treatment of
participants in treatment and control groups
Publication: the tendency to publish only “positive” or
“significant” findings.
8. Reliability
• Refers to the consistency of an
instrument/measurement.
• Thought of as an individual’s “true score” on the
phenomenon you aim to measure minus
“measurement error”
• Two common types of reliability
o Internal consistency: Cronbach’s alpha, KR20
o Inter-Rater: Kappa statistic
• Necessary but not sufficient in determining validity.
10. Validity
Refers to the accuracy of an
instrument/measurement
In other words, “the degree
to which you’re measuring
what you claim to measure”
Two broad types of validity
Internal validity
External validity
11. Internal Validity
Concerns the accuracy of measurement within the
study
Can be threatened by
Biases
Confounding
History: large scale events that change participants’ attitudes or behavior
(e.g., recession)
Maturation: participants change over time, e.g., growth, fatigue etc.
Repeated Testing: participants get wise to the study and remember the
test questions
Compensatory rivalry/resentful demoralization: Control participants work
extra hard to prove themselves or withdraw because not getting
treatment
Diffusion: treatment effects spread from treatment group to control group
12. External Validity
The ability to generalize the findings of your study to
the relevant population.
Threatened by
Bias
Confounding
Non-experimental design (i.e. case-control vs. RCT)
Lack of randomization
External validity is the strongest when a true
experimental design is used.
13. Comparing Research
Designs
Cross-Sectional Studies
Case Control Studies
Cohort Studies
Randomized Control Trial (RCT)
14. Pyramid of Clinical
Evidence
Evidence
Systematic Reviews Summaries
& Meta-analyses
RCT Level 1 Evidence
Cohort
Studies Level 2 Evidence
Cross-Sectional Case Control
Studies: Level 2.3 Studies
Level 3 Evidence
Case Series
Case Reports
Ideas, Editorials, Opinions
Animal research
In vitro (‘test tube’) research
15. Cross-Sectional Studies
“Snapshot” of a population.
People are studied at a “point” in time,
without follow-up.
Strength of evidence…
What are some research questions that can
be answered with cross-sectional designs?
16. Advantages and Disadvantages
of Cross-Sectional Studies
Advantages Disadvantages
Fast and inexpensive Can’t determine
No loss to follow-up causal relationship
Springboard to Impractical for rare
expand/inform diseases
research question Risk for nonresponse
Can target a larger
sample size
17. Case-Control Studies
Always retrospective
Prevalence vs. Incidence
A sample with the disease from a population is
selected (cases).
A sample without the disease from a population is
selected (controls).
Groups are compared using possible predictors of
the disease state.
18. Advantages and Disadvantages
of Case-Control Studies
Advantages Disadvantages
Cannot estimate
High information yield incidence of disease
with few participants
Limited outcomes can
Useful for rare be studied
outcomes
Highly susceptible to
biases
19. Strategies for Sampling
Controls
Population versus hospital/clinic-based controls
Matching
Individual level
Group level
Using 2 or more control groups
20. For Discussion
“How much does a family history of
alcoholism increase the risk of being an
alcoholic?” The PI plans a case-control
study to answer this question.
How should she pick the cases?
How should she pick the controls?
What are some potential sources of bias in the sampling of
cases and controls?
21. Cohort Studies
A “cohort” is a group of individuals who are
followed or traced over a period of time.
A cohort study analyzes an exposure/disease
relationship within the entire cohort.
Groups selected based on exposure to a risk factor.
23. Are U.S. Athletes more likely to win a
Group of gold medal than Chinese athletes at
Interest the 2012 Olympics?
(U.S.)
Follow over the games Compare
Outcomes
Comparison
Group
(China)
24. Prospective versus Retrospective
Cohort Studies
Exposure Outcome
Assessed at the Followed into
Prospective beginning of the the future for
study (present) outcome
Assessed at some Outcome has
Retrospective point in the past already occurred
25. Advantages and Disadvantages
of Cohort Studies
Advantages Disadvantages
Establish population-based Lengthy and costly
incidence May require very large samples
Accurate relative risk Not suitable for rare/long-latency
Temporal relationship inferred diseases
Time-to-event analysis possible Unexpected environmental changes
Used when randomization not Nonresponse, migration and loss-to-
possible follow-up
Reduces biases (selection, Sampling, ascertainment and
information) observer biases
Can study multiple outcomes Changes over time in staff/methods
26. Randomized Controlled
Trial
Considered the “gold standard” by much of the
research community (level 1 evidence)
Blind vs. double blind
Randomization
Cause & effect
27. Designs of RCTs
• Parallel Group Trial: Patients in the same
randomized group throughout the study
• Cross-over Trial: Patient randomly assigned
to one group then crossed over to other
group at some point. Patient serves as own
control – greatly reduces sources of bias
and confounding
• Factorial Trial: Two or more interventions in a
single experiment.
28. Disadvantages of RCT
Designs
Extremely time and
resource demanding
Unethical in many
situations
Poor external validity if
the RCT is too highly
controlled
Difficult to study rare
events
Therapeutic
misconception
29. Material Learned
Common problems in research
design
Confounding
Bias
Reliability
Validity
Observational Research
Designs
Cross-Sectional
Studies
Case Control Studies
Cohort Studies
Experimental Research
Designs
RCT Design
Questions?
30. In Pairs…
Work together to brainstorm an example of how
your topic could be addressed using 1) a Cross-
Sectional design, 2) a case-control design, 3) a
prospective or retrospective cohort design, and an
RCT.
Be prepared to share your responses
Editor's Notes
Most of this presentation will address the observational research designs because those are the ones students will see most often in their own work (most likely). I briefly touch on the common problems in research design as well as a couple of different RCT designs.
The focus is not to give an in depth description of confounding because you will be hitting it on a separate lecture. I just need to introduce a couple of these concepts so describing the strengths and weaknesses of the various designs can be clearer.
Using one of the example findings from this week’s article to illustrate confounding.
Very general introduction to bias. I will be asking the class to think of possible sources of bias before moving to the next slide.
Would love any examples you may have for these biases based on your area of research.
I’m using a social science version of these constructs, but from what I see Epi/Med lit uses “precision” and “accuracy” in place of reliability and validity.
Since the Olympics are likely still fresh on everyone’s minds, and I coached/played the sport for many years, I have this example of inter-rater inter-observer reliability as the judging for Olympic gymnastics.My more professional examples are:Multiple judges for professional presentations (we have to calculate it every year for our annual residents’ research day)Multiple physicians looking at the same x-ray or CT (have several stories about those)Multiple scorers on the same essay (e.g. the ACT or GRE writing portion)Doctoral/Master’s committees (while they don’t calculate it, the example still makes sense)
Archery is a classic example of validity, and again I went with the Olympic theme for fun.While there are numerous types of validity, I choose to briefly hit on internal and external as they are most related to experiments/studies themselves.
Several of these have great Public Health implications such as History.
Cross-sectional Studies considered 2.3 by the OBGYN journal I looked at, but it is not technically a “Clinical” study.Double check to see if this is the same pyramid of evidence you see used in Epi.
Can measure attitudes, beliefs, behaviors, personal or family history, genetic factors, existing or past health conditions, or anything else that does not require follow-up to assess.The source of most of what we know about the population.Before moving to the next slide I will be asking for some students to discuss what they think could be drawbacks of these designs.
For each of these study types I like to layout the advantages and disadvantages for students to easily see.
I ran into an incidence v prevalence situation with two students on Wednesday, so hitting it here will be good.The largest yet simplest factor in differentiating case-control from cohort is in the way the groups are selected. That has been the best way for me to describe it to students in the past.
Discuss advantages/disadvantages of each. I have examples of how each would be done.
I would be VERY open to suggestions as to what a more relevant example would be for these students.
Another Olympic example for fun.
I find that people oftentimes get confused on figuring out how these are separated. I usually say that in a retrospective you start in the past and work towards the present while in a prospective you start in the present and work towards the future.
Before showing the slides, have the class brainstorm what some of the possible Pros and Cons to a cohort study could be. After we go through them in class, have them get with a partner to try and think of ways to avoid some of the cons.
This part of the presentation is meant to introduce RCTs, and some of the language associated with them. It is by no means meant to be a all-encompassing presentation on the topic.
Therapeutic misconception: control patients believe they are getting the best treatmentRegarding the external validity comment: I plan to acknowledge that both types of validity are high in RCT designs; however, the more like a laboratory you make an experiment (i.e. the more controlled it is), the less it is reflective of how things work in the real world.