2. • Unit 5
• Quality Criteria in Qualitative Research:
Reliability, Validity, Objectivity, Alternative
Criteria, Criteria for Evaluating the Building of
Theories, Quality Assessment as a Challenge for
Qualitative Research, Triangulation, Analytic
Induction, Generalization in Qualitative Research,
The Constant Comparative Method, Process
Evaluation and Quality Management. (5)
3. Quality criteria for research
• As we have seen, the basic definition of scientific
research is that it is a 'disciplined' inquiry, and
therefore one thing research cannot afford is to
be haphazard or lacking rigour.
• Unfortunately, general agreement about research
quality in scholarly circles stops at the recognition
of its importance; when it comes to specifying
the concrete 'quality criteria' to be applied, the
literature is characterized by a host of parallel or
alternative views and very little consensus.
4. • The discussion of quantitative quality standards is best
into three parts: (a) reliability (b) measurement divided
(c) research validity
• Reliability
• The term reliability comes from measurement theory
and refers to the 'consistencies of data, scores or
observations obtained using elicitation instruments,
which can include a range of tools from standardized
tests administered in educational settings to tasks
completed by participants in a research study.
5.
6. Different potential ways to approach this
• Evaluation built onStandardisation
• One-Size-Fits-All Approach
• Criteria Appropriate to
Researchor Research
Appropriate toCriteria
• How to Assess Research Quality in a
Sensitive Way
7. The formulation of research
‘criteria’ is only one solution to
the problem
8. Reformulating Traditional Criteria
Aim here is to make the whole research
process more transparent
• Exact and Coherent Guidelines
are used
• Interviews are transcribed in
a certain way using specific
methods
• Distinction between verbatim
statements in field notes and
summaries are given
• Reliability of whole process can
be seen in reflexive
documentation
9. Reformulating Traditional Criteria
Validating an InterviewSituation
• Are interviewees given any cause to construct a biased
version of their experiences that doesn’t fit with their
views?
• Script is analysed to look for these specific points
and to see whether the data is reliable
Communication Validation
• Participants can be involved in a secondary research
study to assist in validation ofdata
• Participants can reflect on previous answers
and then better articulate their thoughts
10. Reformulating Traditional Criteria
Procedural Validation
• Validation is looked at for the entire
research process and the different
relationships that take place:• Relationship between what is being observed
and the larger cultural, historical, and
organisational contexts within which the
observations are made
• The relationship among the observer, the
observed, and the setting
• The perspectives that are used to render an
interpretation of any ethnographic data
• The role of the reader in the final product
• The issue of author style to render the
11. Alternative, method-appropriatecriteria
Instead of using traditional metrics, we use those that
are more credible for qualitative work
• Are findings grounded inthe data?
•
• Is sampling appropriate
Are data weighed correctly?
• Are inferences logical?
• Analytic strategies appliedcorrectly?
• Alternative explanations accountedfor?
Is the category structureappropriate?
Can decisions bejustified?
•
•
•
•
What isthe degree of researcher bias?
What strategies were used for increasing
credibility?
12. Evaluating Grounded
Theory Specific suggestions on
evaluating Grounded Theory studies
come from Charmaz (2006) who suggests
breaking this down into four criteria
• Credibility
• Originality
• Resonance
• Usefulness
13. •
•
•
•
•
•
Has the research achieved familiarity with the setting or topic?
Is the data sufficient to merit your claims?
• Think about range, number and depth of observations
contained in thedata
Have you made systematic comparisons between observations
and between categories
Do the categories cover a wide range of empirical observations
Are there strong links between the gathered data and your
argument and analysis?
Has the research provided enough evidence for your claims to
allow the reader to form an independent assessment, and
agree with yourclaims?
Evaluating Grounded Theory Credibility
14. •
•
•
•
Are your categories fresh, do they offer new
insights?
Does your analysis provide a new conceptual
rendering of the data?
What is the social and theoretical significance
of this work? How does your theory
challenge, extend, or refine current ideas,
concepts and practices?
Evaluating Grounded Theory Originality
15. •
•
•
•
Do the categories portray the fullness of the
studied experience?
Have you revealed both liminal and unstable taken-for-
granted meanings
Have you drawn links between larger
collectivities or institutions and individuallives?
Does your grounded theory make sense to your
participants or people who share their circumstances?
Does your analysis offer them deeper insights about their
lives and world?
Evaluating Grounded Theory Resonance
16. •
•
•
•
Does your analysis offer interpretations that people can use
in their everyday worlds?
Do your analytic categories suggest any generic processes?
• If so, have you analysed these processes for
implications?
Can the analysis spark further research in other
substantive areas?
How does your work contribute to knowledge, how does
it contribute to making a better world!?
Evaluating Grounded Theory Usefulness
17. A Framework for Analysing Qualitative
Research Data
Research should be:
• Contributory in advancingwider knowledge or understanding
• Defensible in design by providing a research strategy
• Rigorous in conduct through the systemic collection,
analysis, and interpretation of qualitative data
• Credible in Claim through offering well- founded and
plausible arguments about the significance of the data
generated
18. Standards will only be helpful if
they apply to qualitative
research in general and not to
specific approaches
20. Triangulation is the method of location of a
point from two others of known distance
apart, given the angles of the triangle
formed by the three points
y
°
x
z°
21. Triangulation includes researchers takingdifferent perspectives
on an issue under study or more generally in answering
researchquestions.
These perspectives can be substantiated by using several
methods or theoreticalapproaches,
Triangulation should produce knowledge at different levels,
which means they go beyond the knowledge made possible by
one approach and thus contribute to promoting quality in
research.
22. Types of Triangulation
• Data Triangulation
• The use of different sources of data
• Allows the researcher to reach maximum profit when
using the same methods
• Investigator Triangulation
• The use of different observers or interviewers
• This is not the sharing of work, it is a systemic
comparison of researchers influences on theissue
• Theory Triangulation
• Approaching the data with multiple
perspectives and hypothesis in mind
• Points are put side by side to assess their utility and
power.
• Methodological Triangulation
• Different methods are used to look at the same issue
24. Within-Methods Triangulation
Episodic Interviews
Interviews can be used to understand the everyday knowledge
that peoplehave
Semi-Structured interviews can include narratives where
participants talk at length about specific aspects that they have
encountered
BUT there is the chance for participants to switch from a narrative
discussion to a descriptive, argumentative, or other non-native
form ofpresentation
28. Argumentations
Subjective Definitions
Examples
Repisodes
Narratives ofSituations
Stereotypes
Situational Narratives
basedon different levels of
concreteness
Repisodes regularly occurring
situations, no longer based on a
clear local and temporal
reference
Examples abstracted from
concrete situations, and
metaphor also ranging to clichés
andstereotypes
SubjectiveDefinitions which are
asked for within sessions
Linked to Argumentative
Definitionsof terms, explanations
ofconcepts
Within-Methods Triangulation
29. Between-Methods Triangulation
Method
1
Issue of
Research
Choose at least one method which is specifically suited to
exploring the structural aspects of the problem and at least one
which can capture the essential elements of its meaning to
those involved
Method
2
31. In this module we’ve dealt exclusively with
qualitative work. However, it is possible to
combine qualitative and quantitative
methods in order to increase research quality
throughtriangulation
36. Linking Qualitative and
Quantitative Results
Results may converge, that is, are consistent completely, in
general, by tendency or partially. For example - answers in a
representative survey may match with statements from semi-
structured interviews
Results may be complementary. Interviews can provide
deeper, more detailed explanations to complement results
from a questionnaire
Results may diverge. For example, interviews may produce views
that are different when compared to questionnaires. This would
cause the need for further research.
37. Triangulation of qualitative
and quantitative research is
not per se a quality
indicator for qualitative
research, but it can
contribute to overall quality
39. Positioning Triangulation in the
Research Process
Exploration Data
Collection
Data
Interpretation
Generalisation
40. Questions to ask for selecting a
qualitative research method
I. What do I know about the issue of my study or how
detailed is my knowledge already?
II. How developed is the theoretical or empirical
knowledge in the literature about this issue?
III. Is my interest in more generally exploring the field and the
issue of my study?
IV. What is the background of my study and which methods fit
with this?
V. What do I want to get close to in my study?
I. Personal Experiences of a group of people / social
process
II. Reconstruction of underlying research structures
41. Questions to ask for selecting a
qualitative research method
VI. Do I start with a focused research question right away or do I
start with an unfocused approach in order to develop the
question?
VII. Which aggregate do I want to study: personal experience,
interactions or situations, or bigger entities like organisations or
discourse?
VIII.Is it more the single case I am interested in or the comparison of
various cases?
IX. Which resources (time, money, manpower, skills etc.) are
available for my study?
42. Questions to ask for selecting a
qualitative research method
X. What are the characteristics of the field I want to study and of the
people in it? What can you request of them and what not?
XI. What is the claim of generalisation of my study?
XII. What are the ethical issues to take into account that are
affected by selecting a specificmethod?
43. Decide and reflect carefully
whether you should use
qualitative or quantitative
research
• Why qualitative research?
• Which reasons do you have for
the one or theother?
• What are your expectations
forthe qualitative research
that you are planning?
44. Reflect on the theoretical
background of your knowledge
interest
• What is the impact of your
setting on the research?
• How open and closed is your
access to what you wantto
study?
45. Plan your study carefully, but
allow for reconsidering the
steps and modifying according
to the state of play
• What are the resources
availablefor the study?
• How realistic are the aims of
your research inrelation to the
available resources
• What are necessary and
appropriate shortcuts
x
y
z
a
b
47. Think about whom in the field
you should contact and inform
about your research. Reflect
about the relation to establish to
field subjects
• What can you learn about
your research field and
issues from the way you get
into the field or are
rejected?
48. Think about why you chose
your methods for collecting
data
• Is it a decision for a favourite
method or for habitual reasons?
• What could or would
alternative methods
provide?
• What are the impacts of the
methods you use on your data
and your knowledge?
49. Plan carefully how to document
your data and research
experiences
• How exactly should you write
your notes?
• What are the influences of the
documentation on your research
and on your fieldsubjects?
• What are the impacts of the
documentation on your
methodsof collection and
analysis?
50. Think about the way that you
want to present what you have
experienced in the field and
found in your research
• What are the target audiencesof
your writing?
• What is it mainly you want to
convince them about your
research?
• What is the impact of the format
of your writing on your research
and its findings?
51. Plan how to establish the
quality of your research
• What are the quality criteria
your research should meet?
• How should these
criteriabe realised?
• What is their impact on your
research and your field subjects
or relationships?
52. Generalization
• Generalization, which is an act of reasoning that involves
drawing broad inferences from particular observations, is
widely-acknowledged as a quality standard in quantitative
research, but is more controversial in qualitative research.
• The goal of most qualitative studies is not to generalize but
rather to provide a rich, contextualized understanding of
some aspect of human experience through the intensive
study of particular cases. Yet, in an environment where
evidence for improving practice is held in high esteem,
generalization in relation to knowledge claims merits
careful attention by both qualitative and quantitative
researchers.
• Issues relating to generalization are, however, often ignored
or misrepresented by both groups of researchers.
53. • Generalization - A general statement: a
statement about a group of people or things
that is based on only a few people or things in
the group
• Interpretation - The act or result of explaining
or interpreting something: the way something
is explained or understand
54. • Consideration for Generalization of Results
• sampling plays an important role in research
inasmuch as generalization of research results
are based from the samples of the study
• generalization or conclusion should be drawn
only for the population and circumstances for
which the evidence have been collected.
55. Doing any of these reflect a sense of immaturity and lack of
objectivity on the part of the researcher
• Broad generalizations not adequately
supported by research data should be avoided
• Speculations or assumptions not supported by
findings must be avoided
• Opinionated statements or inclusive and
sweeping statements not supported by
research evidence must also be avoided
56. Interpretation of Data
• According to Calderon & Gonzalez (1993),
interpretation involves making “inferences pertinent to
the research relation” investigated from where
generalization are drawn.
• Interpretation reflects the researcher's own
understanding of the research results which are guided
by logic and reason, established theories, and previous
findings.
• Qualitative data requires understanding, digesting,
synthesizing, conceptualizing, and conceptualizing
descriptions of feelings, behaviours, experiences, and
ideas.
