Research Methods in Education 6th Edition

1. SURVEYS
© LOUIS COHEN, LAWRENCE
MANION & KEITH MORRISON

2. STRUCTURE OF THE CHAPTER
• What is a survey?
• Some preliminary considerations
• Planning a survey
• Low response and non-response, and how to
reduce them
• Survey sampling
• Longitudinal, cross-sectional and trend studies
• Strengths and weaknesses of longitudinal,
cohort and cross-sectional studies
• Postal, interview and telephone surveys
• Internet-based surveys
• Comparing methods of data collection in surveys

3. KEY FEATURES OF SURVEYS
• Gather data on a one-shot basis;
• Represent a wide target population
• Generate numerical, often large scale data;
• Provide descriptive, inferential and explanatory data;
• Manipulate key factors and variables;
• Gather standardized information;
• Ascertain correlations;
• Remove contextual data;
• Gather data from multiple choice, closed questions, test
scores or observation schedules;
• Support or refute hypotheses about the target population;
• They are piloted;
• Make generalizations about, and observe patterns of
response in, the targets of focus;
• Gather data which can be processed statistically.

4. SAME SAMPLE /
COHORT OVER
TIME
TYPES OF SURVEY
LONGITUDINAL
CROSS-
SECTIONAL
TREND/
PREDICTION
STUDIES
SNAPSHOTS OF
DIFFERENT
SAMPLES AT ONE
OR MORE TIMES
SELECTED
FACTORS
CONTINUOUSLY
OVER TIME

5. PLANNING A SURVEY
• Specify the exact purpose of the enquiry:
– Primary topics
– Subsidiary topics
– Specific information requirements
• Specify the population on which the survey is
to focus;
• Specify the resources that are available:
– Human (e.g. for administering and processing
survey)
– Material
– Financial
– Administrative
– Temporal
– Geographical
– Software

6. PLANNING A SURVEY
• Data collection
– Researcher-administered
– Self-administered
– Postal survey
– Telephone survey
– Internet survey
– Face-to-face interviews
– Email
• Self-reporting
– May lead to biased reporting

7. PLANNING A SURVEY
• Problem definition
• Sample selection
• Design of measurements
• Concern for participants

8. STAGES IN SURVEY DESIGN
DEFINE OBJECTIVES
DECIDE INFORMATION
SAMPLING INSTRUMENTATION
PILOT
DATA COLLECTION
DATA ANALYSIS
REPORTING
TRAINING

9. STAGES IN CONDUCTING A SURVEY
1. Define the objectives;
2. Decide the kind of survey required;
3. Formulate research questions or hypotheses;
4. Decide the issues on which to focus;
5. Decide the information that is needed to address the
issues;
6. Decide the sampling required;
7. Decide the instrumentation and the metrics required;
8. Generate the data collection instruments;
9. Decide how the data will be collected;
10. Pilot the instruments and refine them;
11. Train the interviewers (if appropriate);
12. Collect the data
13. Analyze the data;
14. Report the results.

10. PROBLEMS IN SURVEYS
• Poor sampling
• Poor question design and wording (failure to
operationalize)
• Incorrect or biased responses
• Low response or non-response

11. HOW TO IMPROVE RESPONSE RATES
• Identify where the problem lies:
– The survey never reached the intended people;
– People refuse to answer ;
– People may not be available;
– People may not be able to answer the questions;
– People may not actually have the information
requested;
– People may overlook some items in error;
– The survey was completed and posted but failed to
return.
– The pressure of competing activities on the time of
the respondent;
– Potential embarrassment at their own ignorance if
respondents feel unable to answer a question;

12. HOW TO IMPROVE RESPONSE RATES
– Ignorance of the topic/no background in the topic;
– Dislike of the contents or subject matter of the
interview;
– Fear of possible consequences of the survey to
himself/herself or others;
– Lack of clarity in the instructions;
– Fear or dislike of being interviewed;
– Sensitivity of the topic, or potentially insulting or
threatening topic;
– Betrayal of confidences;
– Losing the return envelope or return address;
– The wrong person may open the mail, and fail to
pass it on to the most appropriate person.

13. HOW TO IMPROVE RESPONSE RATES
• Ensure that the topic is interesting and motivating;
• Ensure that it is easy to complete and layout is attractive;
• Include a covering letter that explains the survey;
• Choose a good time to conduct the survey;
• Follow-ups and polite reminders;
• Pre-paid return stamped addressed envelopes;
• Institutional affiliation, survey sponsorship or support from
a high status agent;
• Financial incentives and rewards for return;
• Making instructions about responses and return very
clear;
• Avoid open-ended questions unless these are really
important;
• Avoid placing open-ended questions at the start of a
questionnaire;

14. HOW TO IMPROVE RESPONSE RATES
• Have easy-to-follow instructions and spacing of
the text;
• Flatter the participants without being seen to
flatter them;
• Providing information about the research through
a covering letter and/or advance notification;
• Making the survey look very unlike junk mail;
• Consider asking the respondents for an interview
to complete the survey questionnaire;
• Deliver the questionnaire personally rather than
through mail;
• Ensure that the questions or items are non-
judgemental;
• Assure confidentiality and anonymity.

15. HOW TO IMPROVE RESPONSE RATES
• Follow a sequence:
– Send non-respondents a friendly reminder
after ten days, stressing the importance of
the research;
– Send a further friendly reminder ten days
after the initial reminder, stressing the
importance of the research;
– Make a telephone call to the respondents
shortly after the second reminder,
indicating the importance of the research.

16. CONSIDERATIONS IN SAMPLING
• A probability and non-probability sample;
• Confidence level and interval;
• The desire to generalize, and to whom;
• The sampling frame;
• The sample size;
• The representativeness of the sample;
• Access to the sample;
• Anticipated response rate.

17. LONGITUDINAL DATA ARE
AFFECTED BY . . .
• History (events occurring may change the
observations of a group under study);
• Maturation (participants mature at different speeds
and in different ways);
• Testing (test sensitization may occur – participants
learn from exposure to repeated testing/interviews);
• The timing of cause and effect: some causes may
produce virtually instantaneous effects and others
may take a long time for the effects to show;
• The direction of causality not always being clear or
singular.
• Comparability of data over time;
• Attrition (dropout);
• Respondents’ memory and recall capabilities.

18. MEMORY IS AFFECTED BY . . .
• The time that has elapsed since the event took
place;
• The significance of the event for the participant;
• The amount of information required for the study
– the greater the amount, the harder it is to
provide;
• The contamination/interference effect of other
memories of a similar event (i.e. the inability to
separate similar events);
• The emotional content or the social desirability of
the content;
• The psychological condition of the participant at
interview.
• Hindsight.

19. ADVANTAGES OF POSTAL SURVEYS
• Reach many people
• Comparatively cheap
• Can be completed at respondents’ preferred
time
• No risk of interviewer bias
• Can reach scattered populations
• Can gather sensitive data (as nobody else
present)

20. DISADVANTAGES OF POSTAL
SURVEYS
• Low response rate
• No check on understanding
• No check whether all items are completed
• Need to have a very simple format
• Presentation matters affect completion

21. ADVANTAGES OF INTERVIEW
SURVEYS
• Improves response rates
• Can clarify queries from respondents
• Can stimulate the respondent to give full
answers
• Can be flexible (e.g. in item sequence)
• Benefit from non-verbal communication
• Can build in trust and rapport
• Ensures that only the respondent answers the
questions

22. DISADVANTAGES OF INTERVIEW
SURVEYS
• Interviewer characteristics affect responses
• Conduct of interview affects responses
• Flexibility can reduce standardization
• Costly in time, travel and training

23. ADVANTAGES OF TELEPHONE
SURVEYS
• Reduce bias in the researcher or the interviewee
• Reduce costs of time and travel
• Easy to find more people to contact

24. DISADVANTAGES OF TELEPHONE
SURVEYS
• Not everyone has a telephone
• Not everyone is available for interview
• People are ex-directory
• The person answering the call may not be the
most suitable person
• Multiple choice and rating scale questions are
difficult
• Order effects can be strong
• People may lie or hang up if questions are
unwelcome, sensitive, too long, too many
• Lack of non-verbal cues

25. ADVANTAGES OF INTERNET-BASED
SURVEYS
• Many people can be reached
• Cheap to administer
• Quick to process results
• Can reach difficult populations
• Anonymity and non-traceability
• Novelty value
• Respondents can complete the questionnaire from
home;
• Respondents can complete it at a time to suit
themselves;
• Respondents can complete the survey over time;
• Complex skip-patterns can be created and
organized by the computer, so that participants do
not have to understand complicated instructions;

26. ADVANTAGES OF INTERNET-BASED
SURVEYS
• The software can prompt respondents to complete
missed items or to correct errors;
• Computer can check incomplete or inconsistent
replies;
• For each screen, the computer can provide an on-
screen indication of how much of the questionnaire
has been completed;
• Reduction of researcher effects;
• Human error is reduced in entering and processing
data;
• Additional features may make the survey attractive;
• Greater generalizability may be obtained as Internet
users come from a wide and diverse population;
• Greater authenticity of responses may be obtained.

27. DISADVANTAGES OF INTERNET-
BASED SURVEYS
• Not everyone has internet access
• Limited control over sampling
• Ethics (e.g. traceability)
• Hardware and software problems
• Expertise of respondents
• Visual presentation takes on added signficance
• Limited number of lines per screen
• People give a minimal response
• Order effects (early responses affect later ones)
• People stop if it is too long or complicated