Best Practices for Cognitive Interviewing
•Transferir como PPTX, PDF•
1 gostou•656 visualizações
The Total Quality Framework (TQF) offers a conceptual foundation from which qualitative researchers are able to think about the quality of their cognitive interviewing designs. The TQF is rooted in the belief that cognitive interviewing and all qualitative research methods must be: credible, analyzable, transparent, and useful.
Recomendados
Recomendados
Mais conteúdo relacionado
Mais procurados
Mais procurados (20)
Semelhante a Best Practices for Cognitive Interviewing
Semelhante a Best Practices for Cognitive Interviewing (20)
Mais de Roller Research
Último
Último (20)
Best Practices for Cognitive Interviewing
- 1. A Best Practices Approach to Cognitive Interviewing Margaret R. Roller, MA Karen Kellard, MA July 18, 2019
- 2. July 18, 2019 1A Best PracticesApproach toCI – Roller & Kellard What is Qualitative Research?Overview Qualitative research & cognitive interviewing Quality in qualitative research & CI Total Quality Framework (TQF)
- 3. July 18, 2019 2A Best PracticesApproach toCI – Roller & Kellard What is Qualitative Research?Qualitative Research & Cognitive Interviewing Attributes Absence of absolute “truth” Importance of context Importance of meaning Participant-researcher relationship Researcher skill set Flexibility of design Types of issues & questions Thematic analysis Researcher as instrument Unique online & mobile capabilities
- 4. July 18, 2019 3A Best PracticesApproach toCI – Roller & Kellard What is Qualitative Research?Qualitative Research & Cognitive Interviewing Attributes Absence of absolute “truth” Importance of context Importance of meaning Participant-researcher relationship Researcher skill set Flexibility of design Types of issues & questions Thematic analysis Researcher as instrument Unique online & mobile capabilities
- 5. July 18, 2019 4A Best PracticesApproach toCI – Roller & Kellard Qualitative Research & Cognitive Interviewing Challenges Importance of meaning Participant-researcher relationship Researcher skill set Flexibility Maximize Data Integrity
- 6. July 18, 2019 5A Best PracticesApproach toCI – Roller & Kellard Strategies & criteria for thinking about quality in qualitative research are not new Various post-hoc criteria for “trustworthiness,” validation & verification strategies, & guidelines (Brinkman & Kvale, 2015; Creswell, 2013; Levitt et al., 2017; Lincoln & Guba, 1985; Maxwell, 2013; Meyrick, 2006) A quality approach specific to cognitive (CI) has also been recognized “…cognitive interviewing practices sometimes lack appropriate scientific rigor…cognitive interviewers should strive for common standards…” (Willis, 2005, p. 134) Quality in Qualitative Research & CI
- 7. July 18, 2019 6A Best PracticesApproach toCI – Roller & Kellard Total Quality Framework Develops critical thinking skills among researchers (& users of the research) by showing how to give explicit attention to quality issues at each stage of the research process regardless of the qualitative method “A useful tool…to (a) gather high-quality data, (b) lead to more robust and valid interpretations of the data, and (c) ultimately generate highly useful outcomes.” (Roller & Lavrakas, 2015, p.16)
- 8. July 18, 2019 7A Best PracticesApproach toCI – Roller & Kellard Total Quality Framework
- 9. July 18, 2019 8A Best PracticesApproach toCI – Roller & Kellard What is the TQF?TQF-Credibility (Data Collection)
- 10. July 18, 2019 9A Best PracticesApproach toCI – Roller & Kellard SCOPE Coverage • A cross-section of the population. • Inclusion of diverse geographic areas (as Sample design • Purposive – not convenience or snowball – sampling. – Purposively select a subset of the population. • If working with targeted lists. – Stratify & select across the entire list. Nonresponse • Gaining access to & cooperation from participants. TQF-Credibility (Data Collection)
- 11. July 18, 2019 10A Best PracticesApproach toCI – Roller & Kellard How Many Cognitive Interviews/Rounds to Complete? 1. Did every interview cover every critical question or area? What questions/areas were not adequately discussed with all participants? 4. Do the data obtained thus far effectively move the research to the next step? 7. Does a review of the observers’ & interviewers’ notes or reflexive journals reveal any concerns about objectivity and interpretations of the data?
- 12. July 18, 2019 11A Best PracticesApproach toCI – Roller & Kellard Achieving valid outcomes Content/information • Cognitive interview protocol. Researcher effects • Interviewer bias. • Interviewer inconsistency. Participant effects • Willingness/ability to provide information. DATA GATHERING TQF-Credibility (Data Collection)
- 13. July 18, 2019 12A Best PracticesApproach toCI – Roller & Kellard TQF-Credibility (Data Collection) Five Key Qualitative Interviewer Skills Build rapport Actively listen Maintain sensitivity to verbal & nonverbal cues Stay focused on the objectives, yet Be flexible so as to not miss new learning
- 14. July 18, 2019 13A Best PracticesApproach toCI – Roller & Kellard TQF-Credibility (Data Collection) Probing Skills WHEN Concurrent Retrospective HOW Proactive Reactive WHAT Comprehension Recall Judgment Response
- 15. July 18, 2019 14A Best PracticesApproach toCI – Roller & Kellard Evokes rationality. Researchers are in essence asking participants to justify their attitudes and behavior. In contemplating a justification, participants seek a response that “makes sense” or is otherwise deemed appropriate. Confuses. “Why did this question make sense to you?” may be difficult and confusing to answer compared to “What are the specific aspects of of this question that helped you make sense of it?”. Asks a different question from the one intended. The question “Why do you think this question is being asked?” is essentially a different question than “What do you think is the purpose of this question?” TQF-Credibility (Data Collection)
- 16. July 18, 2019 15A Best PracticesApproach toCI – Roller & Kellard TQF-Credibility (Data Collection) Answers off-topic, answers a different question Answers are too detailed, gives too much information Answers are too short, too abrupt, lack detail Fast talker, interviewer can’t keep up Social desirability responding Shy participant &/or uneasy with the CI environment Uncomfortable with the think-aloud technique Types of Participants
- 17. July 18, 2019 16A Best PracticesApproach toCI – Roller & Kellard What is the TQF?TQF-Analyzability (Analysis)
- 18. July 18, 2019 17A Best PracticesApproach toCI – Roller & Kellard PROCESSING Transcriptions & coding • Are there resources for transcriptions & coding? • Is this the best approach? Making sense of the data, consider: • Problematic areas – Identify & analyze the underlying cause for these problems » Question context – Impact of question(s) that came before • Researcher & participant effects – Consider how these effects potentially introduce error in the data • Vague or unclear responses – Importance of verification strategies TQF-Analyzability (Analysis)
- 19. July 18, 2019 18A Best PracticesApproach toCI – Roller & Kellard VERIFICATION Add meaning & depth to the data to better inform researcher’s interpretations & recommendations Evidence to support/refute findings Types: • Peer debriefings • Triangulation (data, method, investigator) • Deviant cases (outliers, negative cases) • Reflexive journal (assumptions, beliefs, emotional connection, physical environment) TQF-Analyzability (Analysis)
- 20. July 18, 2019 19A Best PracticesApproach toCI – Roller & Kellard What is the TQF?TQF-Transparency (Reporting)
- 21. July 18, 2019 20A Best PracticesApproach toCI – Roller & Kellard “Thick description” of details in the final document Give consumers of the research what they need to: • Understand the sample & the CI process. • Assign usefulness to the interpretations & recommendations. • Evaluate the transferability to other survey contexts. TQF-Transparency (Reporting)
- 22. July 18, 2019 21A Best PracticesApproach toCI – Roller & Kellard Thick Description Details for Cognitive Interviews
- 23. July 18, 2019 22A Best PracticesApproach toCI – Roller & Kellard What is the TQF?TQF-Usefulness
- 24. July 18, 2019 23A Best PracticesApproach toCI – Roller & Kellard What is the TQF?TQF-Usefulness Do the outcomes Ultimately, reduce survey measurement error? Confirm or deny what is known? Enable researchers to transfer the learning to comparable contexts &/or types of participants?
- 25. July 18, 2019 24A Best PracticesApproach toCI – Roller & Kellard What is the TQF? The Total Quality Framework
- 26. July 18, 2019 25A Best PracticesApproach toCI – Roller & Kellard What is the TQF? CI & qualitative research share unique attributes. The important challenge is how to bring greater rigor to CI without stifling these unique attributes. The Total Quality Framework (TQF) helps develop critical thinking skills re: best practices Central to the TQF is the idea that our research must be credible, analyzable, transparent, & ultimately useful. Not all best practices may be realistic for CI but the TQF offers a tool to give explicit attention to quality issues. Summary
- 27. Thank You! Please feel free to contact us Margaret R. Roller, MA Independent Consultant rmr@rollerresearch.com www.rollerresearch.com Karen Kellard, MA The Social Research Centre Melbourne AUSTRALIA karen.kellard@srcentre.com.au www.srcentre.com.au
- 28. July 18, 2019 27A Best PracticesApproach toCI – Roller & Kellard Reference: Text Roller & Lavrakas, 2015. New York: Guilford Press. http://bit.ly/TQFbook
- 29. July 18, 2019 28A Best PracticesApproach toCI – Roller & Kellard Reference: Blog http://www.researchdesignreview.com https://researchdesignreview.com/tag/total-quality- framework/
- 30. July 18, 2019 29A Best PracticesApproach toCI – Roller & Kellard Brinkmann, S., & Kvale, S. (2015). Interviews: Learning the craft of qualitative research interviewing (3rd ed.). Thousand Oaks, CA: Sage. Creswell, J. W. (2013). Qualitative inquiry & research design (3rd ed.). Thousand Oaks, CA: Sage Publications. Levitt, H. M., Motulsky, S. L., Wertz, F. J., Morrow, S. L., & Ponterotto, J. G. (2017). Recommendations for designing and reviewing qualitative research in psychology: Promoting methodological integrity. Qualitative Psychology, 4(1), 2–22. Lincoln, Y. S., & Guba, E. G. (1985). Naturalistic inquiry. Beverly Hills, CA: Sage Publications. Maxwell, J. A. (2013). Qualitative research design: An interactive approach (3rd ed.). Thousand Oaks, CA: Sage Publications. Meyrick, J. (2006). What is good qualitative research? A first step towards a comprehensive approach to judging rigour/quality. Journal of Health Psychology, 11(5), 799–808. Roller, M. R., & Lavrakas, P. J. (2015). Applied qualitative research design: A total quality framework approach. New York: Guilford Press. Willis, G. B. (2005). Cognitive interviewing: A tool for improving questionnaire design. Thousand Oaks, CA: Sage Publications. ReferencesReferences
Notas do Editor
- These unique attributes pose a challenge. The challenge is…
- How do we integrate these unique facets of CI with a quality approach to the research design in order to arrive at valid data? And that is important because I would suggest that CI ultimately mitigates survey measurement error to the extent that it is done well.
- What I am suggesting is not new.
- What I am presenting today is the TQF It is not prescriptive but rather a way to…. Today, I am asking you to think about many different issues relevant to data integrity & the validity of your outcomes. I realize that some of these best practices may fall outside the scope of your reality when conducting CI BUT they are something to think about – to question yourself – related to these issues of quality. My comments today relate to the face-to-face (in-person or webcam) CI mode, not web probing.
- 9
- One of the considerations within Scope: How many interviews should I complete? I know, CI typically uses small samples (5-15 or more) & are conducted in a short time frame…HOWEVER, these Qs GO BEYOND WHETHER YOU ARE IDENTIFYING NEW PROBS.
- 11
- RESEARCHER & PARTICIPANT EFFECTS are mitigated by using a qualitative interviewer – not survey field interviewer – who is trained on interviewing skills such as
- Mitigating RESEARCHER & PARTICIPANT EFFECTS also has to do with the interviewer’s probing skills. Probing uncovers the underlying meaning of the participant’s reactions/comments. In the broadest, simplest sense, probing skills boil down to: When How and What WHEN: Concurrent, in-the-moment probing is a best practice because it mitigates the participant burden & error associated with memory/recall, and is more conversational which helps build rapport which is a key ingredient in the participant-researcher relationship mentioned earlier HOW: Proactive probes + with scripted AND spontaneous questions are a best practice. + scripted probes bring a systematic approach to CI which adds consistency across interviews which fosters an apples-to-apples comparison at the end. - But error because the anticipated probes may miss important input. + spontaneous probes account for unanticipated behavior or comments, gives interviewer FLEXIBILITY WHAT Comprehension: “WHAT IS THIS Q ASKING?” What is your understanding of this Q? We are asking the participant to paraphrase. Be sure interviewer understands the meaning of the paraphrase & how participant derives meaning, e.g., Q wording, structure (format), & ordering (context). Recall: The probe “Is this Q easy or difficult to answer?” is NOT the right probe. That is NOT what you want to know. The participant’s response is based on an unintended understanding of the Q. Learn from “How did you arrive at your answer?”
- And be careful how you write probing Qs….
- Another important skill of the qual interviewer is the ability to recognize & handle a variety of participant types….. This last bit re: the think-aloud technique speaks to the importance of allowing practice prior to the onset of the CI to reduce participant burden. The shy participant situation also highlights the power imbalance that can occur in the one-on-one interview environment. In addition, there is also the participant who becomes fixated on giving an answer rather than their understanding of what the Q is asking, needing the interviewer to rebrief or redirect the P.
- Resources = Time & money Best approach = Is it the most efficient approach? What will be lost? Gained? Will you lose context & meaning? Making sense–Have we identified all of the problematic areas in the items being tested AND the root causes of problems Across Qs: Not always testing a full set of questions,. May be more evident in piloting? However, in the Debt Stress Index questions we tested two orders – asking about Debt first, and then Debt Stress, and asking about Debt Stress first and then Debt (as we knew from the survey work that people had different acknowledgement of debt depending on the order.
- 18
- Resources may preclude the use of thick description – and it may not be needed – BUT a best practice is to provide details of the CI process so that users of the CI results can understand what you did and determine their confidence in the results and therefore its usefulness and its applicability to other similar survey work. BTW, having clients/sponsors actually attend and observe the CIs can be effective in helping them understand what you communicate in the report.
- It will differ for each study & situation. Failure to interview all sampled: Replace with similar because the sample is so small there is too much of risk we miss stuff by not having them included
- All of this serves to define the Usefulness of your research.
- Does my research..... Help to reduce survey measurement error Confirm or deny what we already knew about the strengths/weaknesses of the survey design & participants Allow us to use this new knowledge in other survey contexts & comparable participants
- To meet this challenge, THE TQF IS NOT PRESCRIPTIVE, RATHER IT FOSTERS ..... For each phase of the research process – Data collection, analysis, reporting, & outcomes