2. Benefits of research to whom
As a graduate student...
To be able to read and understand the
empirical literature in your field; to become a
critical consumer of information.
As a graduate student preparing for a
thesis or dissertation…
To be able to both design and implement your
thesis or dissertation as well as future studies
that interest you.
3. Benefits to whom
As a future practitioner…
To be able to intelligently participate in
research projects, evaluations, and studies
undertaken by your institution.
As an educated citizen ...
To understand the difference between
scientifically acquired knowledge and other
kinds of information.
4. What Research Is Not
Research isn’t information gathering:
Gathering information from resources such books or
magazines isn’t research.
No contribution to new knowledge.
Research isn’t the transportation of facts:
Merely transporting facts from one resource to
another doesn’t constitute research.
No contribution to new knowledge although this
might make existing knowledge more accessible.
5. What Research Is
Research is:
“…the systematic process of collecting and
analyzing information (data) in order to increase
our understanding of the phenomenon about
which we are concerned or interested.”1
6. What’s the Difference Between
“Method” and “Methodology”?
Method: Methodology:
Techniques for The underlying theory
gathering evidence and analysis of how
The various ways of research does or
proceeding in should proceed, often
gathering information influenced by
discipline
7. Epistemology, Methodology, and
Method
“a research method is a technique for (or
way of proceeding in) gathering evidence"
while
"methodology is a theory and analysis of
how research does or should proceed" and
"an epistemology is a theory of
knowledge"
8. "It is the theory that decides
what can be observed."
Albert Einstein
9. Research Characteristics
1. Originates with a question or problem.
2. Requires clear articulation of a goal.
3. Follows a specific plan or procedure.
4. Often divides main problem into subproblems.
5. Guided by specific problem, question, or hypothesis.
6. Accepts certain critical assumptions.
7. Requires collection and interpretation of data.
8. Cyclical (helical) in nature.
10. Research Projects
Research begins with a problem.
This problem need not be Earth-shaking.
Identifying this problem can actually be the
hardest part of research.
In general, good research projects should:
Address an important question.
Advance knowledge.
11. Research Project Pitfalls
The following kinds of projects usually don’t
make for good research:
Self-enlightenment.
Comparing data sets.
Correlating data sets.
Problems with yes / no answers.
12. High-Quality Research
(1 of 2)
Good research requires:
The scope and limitations of the work to be clearly
defined.
The process to be clearly explained so that it can be
reproduced and verified by other researchers.
A thoroughly planned design that is as objective as
possible.
13. High-Quality Research
(2 of 2)
Good research requires:
Highly ethical standards be applied.
All limitations be documented.
Data be adequately analyzed and explained.
All findings be presented unambiguously and all
conclusions be justified by sufficient evidence.
14. Sources of Research Problems
Observation.
Literature reviews.
Professional conferences.
Experts.
15. Stating the Research Problem
Once you’ve identified a research problem:
State that problem clearly and completely.
Determine the feasibility of the research.
Identify subproblems:
Completely researchable units.
Small in number.
Add up to the total problem.
Must be clearly tied to the interpretation of the data.
16. Research
Acquisition of Knowledge
Knowledge
v/s
Information
(Theoretically, concerned with developing,
exploring, or testing theories)
17. Theory
What exists?
Why exists?
What will happen in future?
18. How to acquire Knowledge?
Inductive Reasoning
( works moving from specific
observation to broader
generalisation, bottom approach)
Deductive Reasoning
( more general to more specific or
top down approach)
21. Positivism
Goal of Knowledge is to describe
the phenomena that are
experienced,
There is interdependence of
observation and theory, our
observations are theory laden
22. Scientific
thought
Francis Bacon
Rene Descartes
John Stuart
Karl Popper
Thomas Kuhn
Feyer bend
Steven Hagen
23. Hypotheses
Hypotheses are tentative, intelligent guesses as
to the solution of the problem.
There is often a 1-1 correspondence between a
subproblem and a hypothesis.
Hypotheses can direct later research activities since
they can help determine the nature of the research
and methods applied.
24. Delimitations
All research has limitations and thus certain
work that will not be performed.
The work that will not be undertaken is
described as the delimitations of the research.
25. Definitions
Define each technical term as it is used in
relation to your research project.
This helps remove significant ambiguity from the
research itself by ensuring that reviewers, while they
may not agree with your definitions, at least know
what you’re talking about.
26. Assumptions
Assumptions are those things that the
researcher is taking for granted.
For example: a given test instrument accurately and
consistently measures the phenomenon in question.
As a general rule you’re better off documenting
an assumption than ignoring it.
Overlooked assumptions provide a prime source of
debate about a research project’s results.
27. Importance of the Study
Many research problems have a kind of
theoretical feel about them. Such projects often
need to be justified:
What is the research project’s practical value?
Without this justification, it will prove difficult to
convince others that the problem in question is
worth study.
28. Research Proposals
Research proposals are documents that describe
the intended research including:
Problem and subproblems.
Hypotheses.
Delimitations.
Definitions.
Assumptions.
Importance.
Literature review.
29. Literature Review
A literature review is a necessity.
Without this step, you won’t know if your problem
has been solved or what related research is already
underway.
When performing the review:
Start searching professional journals.
Begin with the most recent articles you can find.
Keep track of relevant articles in a bibliography.
Don’t be discouraged if work on the topic is already
underway.
30. Literature Review Pitfalls
(1 of 2)
Be very careful to check your sources when
doing your literature review.
Many trade magazines are not peer reviewed.
Professional conferences and journals often have
each article reviewed by multiple people before it is
even recommended for publication.
The IEEE and ACM digital libraries are good places to
start looking for legitimate research.
31. Literature Review Pitfalls
(2 of 2)
The Internet can be a good source of
information. It is also full of pseudo-science and
poor research.
Make sure you verify the claims of any
documentation that has not been peer reviewed
by other professionals in the computing industry.
33. Research Process
Research is an extremely cyclic process.
Later stages might necessitate a review of earlier
work.
This isn’t a weakness of the process but is part
of the built-in error correction machinery.
Because of the cyclic nature of research, it can
be difficult to determine where to start and
when to stop.
34. Step 1: A Question Is Raised
A question occurs to or is posed to the
researcher for which that researcher has no
answer.
This doesn’t mean that someone else doesn’t already
have an answer.
The question needs to be converted to an
appropriate problem statement like that
documented in a research proposal.
35. Step 2: Suggest Hypotheses
The researcher generates intermediate
hypotheses to describe a solution to the
problem.
This is at best a temporary solution since there is as
yet no evidence to support either the acceptance or
rejection of these hypotheses.
36. Step 3: Literature Review
The available literature is reviewed to determine
if there is already a solution to the problem.
Existing solutions do not always explain new
observations.
The existing solution might require some revision or
even be discarded.
37. Step 4: Literature Evaluation
It’s possible that the literature review has
yielded a solution to the proposed problem.
This means that you haven’t really done research.
On the other hand, if the literature review turns
up nothing, then additional research activities
are justified.
38. Step 5: Acquire Data
The researcher now begins to gather data
relating to the research problem.
The means of data acquisition will often change
based on the type of the research problem.
This might entail only data gathering, but it could
also require the creation of new measurement
instruments.
39. Step 6: Data Analysis
The data that were gathered in the previous
step are analyzed as a first step in ascertaining
their meaning.
As before, the analysis of the data does not
constitute research.
This is basic number crunching.
40. Step 7: Data Interpretation
The researcher interprets the newly analyzed
data and suggests a conclusion.
This can be difficult.
Keep in mind that data analysis that suggests a
correlation between two variables can’t automatically
be interpreted as suggesting causality between those
variables.
41. Step 8: Hypothesis Support
The data will either support the hypotheses or
they won’t.
This may lead the researcher to cycle back to an
earlier step in the process and begin again with a
new hypothesis.
This is one of the self-correcting mechanisms
associated with the scientific method.
42. Common Methodologies
Methodologies are high-level approaches to
conducting research.
The individual steps within the methodology might
vary based on the research being performed.
Two commonly used research methodologies:
Quantitative.
Qualitative.
43. Methodology Comparison
Quantitative Qualitative
Explanation, prediction Explanation, description
Test theories Build theories
Known variables Unknown variables
Large sample Small sample
Standardized instruments Observations, interviews
Deductive Inductive
44. An Overview of Empirical
Research Methods
Descriptive (Qualitative)
Ethnography
Experimental
Case Study
(Quantitative)
Suvey/Sampling
True Experiment
Focus Groups
Discourse/Text Analysis
Quasi-Experiment
Quantitative Description Meta-Analysis
Prediction/Classification
45. Assessing Methods
Research Question(s) is/are key
Methods must answer the research
question(s)
Methodology guides application
Epistemology guides analysis
46. Ethnographies
+ Observational field work done in the actual
context being studied
+ Focus on how individuals interrelate in
their own environment (and the influence
of this environment)
- Difficult to interpret/analyze
- Time consuming/expensive
- Can influence subject behavior
47. Case Studies
+ Focus is on individual or small group
+ Able to conduct a comprehensive analysis
from a comparison of cases
+ Allows for identification of variables or
phenomenon to be studied
- Time consuming
- Depth rather than breadth
- Not necessarily representative
48. Survey Research
+ An efficient means of gathering large
amounts of data
+ Can be anonymous and inexpensive
- Feedback often incomplete
- Wording of instrument can bias feedback
- Details often left out
49. Focus Groups
+ Aid in understanding audience, group, users
+ Small group interaction more than individual
response
+ Helps identify and fill gaps in current knowledge
re: perceptions, attitudes, feelings, etc.
- Does not give statistics
- Marketing tools seen as “suspect”
- Analysis subjective
50. Discourse/Text Analysis
+ Examines actual discourse produced for a
particular purpose (job, school)
+ Helps in understanding of context,
production, audience, and text
+ Schedule for analysis not demanding
- Labor intensive
- Categories often fluid, making analysis
difficult
51. Quantitative Descriptive Studies
+ Isolates systematically the most important
variables (often from case studies) and to
quantify and interrelate them (often via
survey or questionnaire)
+ Possible to collect large amounts of data
+ Not as disruptive
+ Biases not as likely
- Data restricted to information available
52. Discourse/Text Analysis
+ Examines actual discourse produced for a
particular purpose (job, school)
+ Helps in understanding of context,
production, audience, and text
+ Schedule for analysis not demanding
- Labor intensive
- Categories often fluid, making analysis
difficult
53. Quantitative Descriptive Studies
+ Isolates systematically the most important
variables (often from case studies) and to
quantify and interrelate them (often via
survey or questionnaire)
+ Possible to collect large amounts of data
+ Not as disruptive
+ Biases not as likely
- Data restricted to information available
54. Prediction and Classification
Studies
Goal is to predict behaviors:
Prediction forecasts and interval variable
(Diagnostic/TAAS scores)
Classification forecasts a nominal variable (Major
selection after taking 2311)
+ Important in industry, education to predict behaviors
- Need substantial population
- Restricted range of variables can cause misinterpretation
- Variables cannot be added together; must be weighted
and looked at in context of other variables
55. Positive Aspects of Descriptive/
Qualitative Research
Naturalistic; allows for subjects to interact
with environment
Can use statistical analysis
Seeks to further develop theory (not to
influence action); Prescientific
Coding schemes often arise from interplay
between data and researcher’s knowledge
of theory
56. Problems with
Descriptive/Qualitative
Research
Impossible to overlay structure
Impossible to impose control
Subject pool often limited, not
representative
Seen as more “subjective,” less rigorous
Beneficial only in terms of initial
investigation to form hypothesis
57. Experimental Research: True
Experiment
+ Random sampling, or selection, of subjects
(which are also stratified)
+ Introduction of a treatment
+ Use of a control group for comparing subjects
who don’t receive treatment with those who do
- Adherence to scientific method (seen as positive,
too)
- Must have both internal and external validity
- Treatment and control might seem artificial
58. Experimental Research: Quasi-
Experiment
+ Similar to Experiment, except that the subjects
are not randomized. Intact groups are often
used (for example, students in a classroom).
+ To draw more fully on the power of the
experimental method, a pretest may be
employed.
+ Employ treatment, control, and scientific method
- Act of control and treatment makes situation
artificial
- Small subject pools
59. Meta-Analysis
+ Takes the results of true and quasi-experiments
and identifies interrelationships of conclusions
+ Systematic
+ Replicable
+ Summarizes overall results
- C/C apples and oranges?
- Quality of studies used?
60. Positive Aspects of
Experimental Research
Tests the validity of generalizations
Seen as rigorous
Identifies a cause-and-effect relationship
Seen as more objective, less subjective
Can be predictive
61. Problems with Experimental
Research
Generalizations need to be qualified
according to limitation of research
methods employed
Controlled settings don’t mirror actual
conditions; unnatural
Difficult to isolate a single variable
Doesn’t allow for self-reflection (built-in)
62. Testing the Waters
How do you come up with a good
research question?
How do you determine if the method you
plan to use will answer your question?
What epistemology should you use to
analyze data?
63. Quantitative Methods
Samplingst
Testing of Hypothesis
Chi Square Test
ANOVA
Multivariate Analysis
While both of these are tools used during research, they are not sufficient for research.
These don’t represent some kind of linear plan, but are rather common characteristics shared by almost all legitimate research regardless of the venue by which that research was conducted.
The concept of “important” questions is subjective and will depend on who you ask as well as the purpose of the research. For instance, PhD students have a different “bar” than Masters students owing to the requirement that their research be “original and significant.”
The first case is really an exercise in data gathering and doesn’t contribute new knowledge. The same argument holds for the second case. The third case is a mathematical statement but doesn’t address the reasons for the correlation (which might lead to new knowledge). A similar argument applies to the fourth case.
If other researchers can’t confirm your results, you may be faced with having studied an anomaly. Similarly, without a solid plan, you might have inadvertently introduced errors into the experimental design which immediately calls your results into question.
There are very few “perfect” research designs where some flaws aren’t present. That’s normal. However, these flaws must be documented as well as their possible impact on the outcome. While this won’t stop reviewers from criticizing the work, it makes it clear that you are aware of the problems and their impact upon your work.
Many of us have professional experience which can lead to possible research. Always be careful to differentiate between research and self-enlightenment. A lot of computer literature, particularly research journals such as IEEE or ACM, show good research problems and possible sources of future work. Such future work can provide a good starting point for research projects. The same groups also host professional conferences. DePaul has a student chapter of the ACM that might provide a good source of inspiration for research. Many of the faculty here at DePaul have all kinds of problems that they’re trying to solve.
When documenting the proposed research, you should be as precise as you can. You’ll probably find yourself editing and revising many times to attain the necessary level of precision and clarity.
Documenting the delimitations is just as important as documenting the intended research. In essence, the problem and delimitations describe the scope of the project. Keep the delimitations in mind as you move forward. Current delimitations might provide avenues for valuable future work.
The accurate and consistent measurement of some phenomenon is called validity and reliability respectively.
Many times the ability of a researcher to justify the importance of their research topic is directly proportional to their ability to receive funding. This basically requires good salesmanship.
There are some basic variations on this theme, but all of these areas will be covered. This format serves to crystalize your thought process and to help ensure that no critical elements of your research have been neglected.
Some journals include IEEE and ACM. Be careful of trade journals; they’re often not peer reviewed which can call the content into question in terms of its reliability and quality. The bibliography could be a Word document, Excel spreadsheet, or bibliographic database. Even if the article doesn’t directly pertain to your current project, it might provide you with ideas.
We’ll try to give some guidelines as to how to choose starting and stopping points during your research work. Please keep in mind that the following steps don’t have to be slavishly followed in the sequence in which they are presented.
The literature review is often a good source for additional ideas. This is also a good place to go in conjunction with the prior step; there’s no point in wasting good brain cells coming up with hypotheses that solve a problem that has already been adequately addressed.
Keep in mind that just because you didn’t find a solution today, doesn’t mean that one won’t show up tomorrow. This is one of the reasons that researchers are always reading and trying to keep up to date with current trends.
A statement of causality is very difficult to demonstrate because there often many other confounding factors. For an example of this, do a quick bit of reading on the hoops researchers had to go through while trying to show a causal link between smoking and certain kinds of cancer.
The choice of methodology might be governed by the kind of research being conducted. For example, the hard sciences tend to favor quantitative methodologies whereas the social sciences often gravitate toward qualitative approaches. It’s quite common for both methodologies to be used during the course of a single research project.
These are some the differences in the intent and approaches between quantitative and qualitative research. There are other significant differences in approach as well, but these are some of the highlights. Keep in mind that research design is not a simple task.