Brief presentation intended to be a primer on experiments, to be used independently before students look at specific examples.

1. Types of research studies
1
CASE STUDIES
EXPERIMENTS
DESIGN STUDIES
SURVEYS
MITE 6025 Session 2 (van Aalst)9/26/2013

2. What is an experiment?
 A study that tests a hypothesis about
a causal effect using at least two
independent samples
 The experimental sample receives the
intervention of interest
 The control sample does not receive
that intervention but is equivalent in all
other respects
2
2.Experiments
MITE 6025 Session 2 (van Aalst)9/26/2013

3. Hypotheses
 A hypothesis is an educated guess
about what is going to happen during the
experiment
 It usually is based on an implication
(prediction) of a theory
MITE 6025 Session 2 (van Aalst) 39/26/2013
2.Experiments

4. Pre- and post-test
 Having experimental and control
samples not enough
 Also need to test both samples before
(pre-test) and after the treatment
(post-test)
4 MITE 6025 Session 2 (van Aalst)9/26/2013
2.Experiments

5. Simplest experimental design
Experimental Control
• Pretest on
dependent
variable
• Pretest on
other relevant
variables
• Pretest on
dependent
variable
• Pretest on
other relevant
variables
• Treatment • No treatment
• Posttest on
dependent
variable
• Posttest on
dependent
variable
 Generalization to
population
 Random selection
from population
 Random
assignment to
conditions
MITE 6025 Session 2 (van Aalst) 59/26/2013
2.Experiments

6. Sample results for dependent
variable
Experimental Control
Pretest 50.1 (10.2) 48.4 (10.7)
Posttest 72.0 (8.5) 60.9 (10.9)
 The samples are
similar at the
pretest
 The experimental
sample
outperforms the
control sample on
post-test
 Need to calculate
the probability of
getting this effect
MITE 6025 Session 2 (van Aalst) 69/26/2013
2.Experiments

7. Preparation (pretests)
 English proficiency test
 Chinese proficiency test
 General science achievement test
(Chinese)
 Specific science test (same language as
task)
MITE 6025 Session 2 (van Aalst) 79/26/2013
2.Experiments

8. Task design
 A reading task in English and Chinese
 Exactly the same content in both versions
 The same reading level in each language
 Not too long
MITE 6025 Session 2 (van Aalst) 89/26/2013
2.Experiments

9. Finishing up (post-tests)
 Specific science post-test for both
samples
 Time for completion for both samples
MITE 6025 Session 2 (van Aalst) 99/26/2013
2.Experiments

10. Summary
Stage Experimental sample Control sample
Preparation • English proficiency
test
• Chinese proficiency
test
• General science test
(Chinese)
• Specific science test
(English)
• English proficiency
test
• Chinese proficiency
test
• General science test
(Chinese)
• Specific science test
(Chinese)
intervention • Reading task
(English)
• Task completion
time
• Reading task
(Chinese)
• Task completion
time
Finishing up • Specific science test
(English)
• Specific science test
(Chinese)
MITE 6025 Session 2 (van Aalst) 109/26/2013
2.Experiments

11. Sample results
Experimental (n = 30) Control (n = 30)
English test 672 (130) 520 (160)
Chinese test 620 (150) 590 (140)
General science test 50.1 (10.0) 42.8 (10.3)
Specific science pre-test 40.0 (9.5) 35.8 (10.2)
Specific science
posttest
60.1 (9.9) 57.9 (13.2)
Completion time 17.5 (3.5) min 25.7 (3.9) min
MITE 6025 Session 2 (van Aalst) 119/26/2013
2.Experiments

12. Extensions
 More than one treatment
 More dependent variables
 More control variables
MITE 6025 Session 2 (van Aalst) 129/26/2013
2.Experiments

13. Quasi-experiments
 When used
 random selection and/or random assignment
are impossible
 Sample equivalence is difficult to achieve
 E.g. the use of in-tact classes
 Assignment to classes not random
 One class does the treatment, the other class
does not
 Classes may be taught by different teachers
MITE 6025 Session 2 (van Aalst) 139/26/2013
2.Experiments