Non-experimental designs are explained. Threats to validity are defined and exemplified. Tips to deal with these threats are provided.

1. Exploiting Rapid
Change in Technology
Enhanced Learning
… for Post Graduate Education
Quantitative Methodology Part Two

2. Ethical Research

3. Our Focus for Today
1. Discussion of the exercises presented in part one
2. Pre-experimental quantitative research methods
3. Threats to validity

4. Exercises from Part One

5. Establish what research design is appropriate for
this study…
•Problem: A researcher has three different groups of employees
(engineers, administrators, sales personal) which are not
performing at the expected level. Data shows three different levels
of current performance: high, average, and low.
•Purpose: To determine the effects of different types of feedback
on employees’ performance
•Research Question: What is the effect of different types of
feedback over the performance of employees?
• Values of the independent variable: positive, negative, no
feedback

6. Establish what research design is appropriate for
this study…
•Problem: Faculty morale is low at a university according to
the results of a questionnaire applied.
•Purpose: To establish the effect of sensitivity training on
faculty morale
•Research Question: What is the effect of sensitivity training
workshops on faculty morale?
• Values of the independent variable: treatment, no
treatment

8. Establish what research design is appropriate for
this study…
•Problem: A new calculus program has been suggested to
improve students performance
•Purpose: To establish the effect of the suggested program
on students’ calculus scores
•Research Question: What is the effect of the suggested
calculus program on students’ scores?
• Values of the independent variable: treatment, no
treatment

9. Now do the same with your own study…
•Problem:
• Research Question:
• Research Design:
•Research Method:

10. Sharing Your Experience
1. What was the most clear point?
2. What was the least clear point?
3. What would you like to say about the experience?
4. Any specific exercise you want to talk about?

11. Pre-Experimental Designs
• Ex Post Facto Research
• Survey Design

12. Ex Post Facto Research
• Description: Variables are studied after the fact has occurred without
interference from the researcher. Possible relationships and effects are
searched. It is applied in contexts in which it is not possible or acceptable
to manipulate the characteristics of human participants or in situations in
which it is not practical or ethically acceptable to apply the full protocol of
a true experimental design. They are widely used in social sciences.
• Types:
– Causal-Comparative Design
– Correlational Design

13. Causal-Comparative Design
• Description: Researchers try to establish the possible cause-effect
relationship that already exists between variables in a
nonexperimental setting. There are comparison groups rather than
experimental and control groups.
• Basic approaches:
– Retrospective
– Prospective

14. Correlational Design
• Description: It determines whether or not two variables are correlated,
which means that an increase or decrease in one variable corresponds to
an increase or decrease in the other variable. It is very important to note
that correlation does not imply causation, just an association.
• Types :
– Relational Design
– Prediction design

15. Survey Design
• Description: It intends to describe a situation using the answers
to questions
• Types of Designs:
– Longitudinal Design
• Trend Study
• Cohort Study
• Panel Study
– Cross-Sectional Design
• Types of Surveys:
– Questionnaire
– Interview

16. Difference-in-Difference Estimation
The simplest set up is one where outcomes are observed for two groups
for two time periods. One of the groups is exposed to a treatment in the
second period, but not in the first period. The second group is not
exposed to the treatment during either period. Then, the researcher
estimates the “difference of differences”
An estimator, not a design. Used to study policy questions by estimating
treatment effects with nonexperimental data

18. Example
A study about the effect of the higher minimum wage in fast-food restaurants
in Pennsylvania (where the wage is constant) and New Jersey (where the wage
has been increased). These are some of the results:
State Before Increase After Increase Difference
New Jersey
(treatment)
20.44
(0.51)
21.03
(0.52)
0.59
(0.54)
Pennsylvania
(control)
23.33
(1.35)
21.07
(0.94)
-2.16
(1.25)
Difference -2.89
(1.44)
-0.14
(1.07)
2.76
(1.36)
Note: Standard errors in parentheses
Taken from: Albouy (n.d)

19. Taken from: Wiersma, W. & Jurs, S.G. (2009)
Threats to Internal Validity

20. History
Description: Unanticipated events occurring while the
experiment is in progress that affect the dependent
variable

21. Maturation
Description: Processes operating within the subjects as
a function of time

22. Testing
Description: The effect that taking one test has on the
results of a subsequent test

23. Instrumentation
Description: An effect due to inconsistent use of the
measuring instruments, observers, or scorers that may
affect the results

24. Statistical Regression
Description: An effect caused by a tendency for subjects
selected on the basis of extreme scores to regress
toward an average performance on a subsequent test. It
is also known as “regression to the mean”

25. Selection
Description: An effect due to the groups of subjects
not being randomly assigned to groups; a selection bias
is operating such that the groups are not equivalent

26. Mortality
Description: An effect due to subjects dropping out of
the experiment. The subjects that stay may be more
motivated or capable and that affects the results

27. Selection- Maturation Interaction
Description: An effect of maturation not being
consistent across the groups because of some selection
factor may lead to confusing results and an erroneous
interpretation of the effect of the treatment.

28. Taken from: Wiersma, W. & Jurs, S.G. (2009)
Threats to External Validity

29. Interaction Effect of Testing
Description: Pretesting interacts with the experimental
treatment and causes some effect such that the results
cannot be not generalized to an unpretested population

30. Interaction Effects of Selection
Biases and the Experimental
Treatment
Description: An effect of some selection factor of intact
groups interacting with the experimental treatment that
would not be the case if the groups were formed
randomly

31. Reactive Effects of Experimental
Arrangements
Description: An effect that is due to the artificial or novel
experimental setting. It may also threaten internal
validity

32. Multiple-Treatment Interference
Description: When the same subjects receive two or more
treatments, there may be a carryover effect between
treatments and therefore the results cannot be
generalized to single treatments

33. Taken from: Wiersma, W. & Jurs, S.G. (2009)
Threats to Construct Validity

34. Inadequate Preoperational
Explication of Constructs
Description: Insufficient definition of the variables

35. Mono-Operation Bias
Description: Only one form of the experimental variable is
implemented

36. Mono-Method Bias
Description: Only one form of the dependent variable is
implemented

37. Hypothesis-Guessing within
Experimental Conditions
Description: Participants behave differently when they
know they are part of an experiment
Participants’ behavior can also threaten internal validity

38. Confounding Constructs and Levels
of Constructs
Description: Drawing conclusions about variables when
some levels of the variable are absent

39. Taken from: Wiersma, W. & Jurs, S.G. (2009)
Threats to Statistical Conclusion
Validity

40. Low-Statistical Validity
Description: Using a sample size that is too small to detect
differences between groups

41. Violated Assumptions of Statistical
Tests
Description: Failing to meet the underlying assumptions

42. Fishing and the Error Rate Problem
Description: Capitalizing on chance findings

43. Reliability of Measures
Description: Using technically inadequate measures

44. Now you know
1. The pre-experimental designs
2. What difference-in-difference
estimation is
3. Different types of threats to validity

45. Reference
Albouy, D. (n.d). Program evaluation and the difference in
difference estimator. Retrieved from
http://eml.berkeley.edu/~webfac/saez/e131_s04/diff.pdf
Wiersma, W. & Jurs, S.G. (2009). Research methods in education:
An introduction (9th ed.). Boston, MA: Allyn and Bacon

46. What’s Up at MN & DN this summer?
1. VERY IMPORTANT – backwards map your summer work
2. 30 day writing challenge, 30 day work-life balance challenge, and 365s -
keep you in touch with your work
3. Group work – RLC writing OR
Lingerers

47. Avoid this one
1. VERY IMPORTANT – backwards map your summer work
2. 30 day writing challenge, 30 day work-life balance challenge, and 365s -
keep you in touch with your work
3. Group work – RLC writing OR Lingerers