1. A Pilot Study: What, Why, and How
Soleh Ayubi
09-20-2012
1

2. Until 2012, no one ever published trials
paper related to social media…
2

3. Until 2012, no trials paper related to social
media are available…
2006 2007 2008 2009 2010 2011 2012
Writing
papers
Gathering
ideas and
writing
proposals
Submitting
proposals
and
waiting for
the results
Get a
funding,
Administra
tive
process,
consolidat
e teams,
etc.
Preparing
the study:
IRB,
logistics,
recruitment
, etc.
Trials
3

4. What is a Pilot Study?
 A pilot projector study as an experimental,
exploratory, test, preliminary, trial or try out
investigation (Concise Oxford Thesaurus, 2002).
 “ ...test of the methods and procedures to be
used on a larger scale if the pilot study
demonstrates that the methods and procedures can
work” (A Oxford Dictionary of Epidemiology, 2001)
 “...investigation designed to test the feasibility of
methods and procedures for later use on a large
scale or to search for possible effects and
associations that may be worth following up in a
subsequent larger study” (Everitt B, 2006).
4

5. What is a Pilot Study?
 Synonymous with a feasibility study (Thabane. L, et
al., 2010).
 The main goal of pilot studies is to assess feasibility
so as to avoid potentially disastrous consequences
of embarking on a large study - which could
potentially “drown” the whole research effort
(Thabane. L., 2010).
 Pilot studies are required to do prior to full-scale
phase III trials (G.A. Lancester, et al., 2004).
5

6. Reasons for Pilot Study
 Major funding bodies (UK Medical Research Council
and NIH) require evidences from a pilot study before
allocating fund for RCTs (G.A. Lancester, et al.,
2004).
 It permits preliminary testing of the hypotheses that
leads to testing more precise hypotheses in the main
study. It may lead to changing some hypotheses,
dropping some, or developing new hypotheses (G.A.
Lancester, et al., 2004; Thabane. L., 2010).
 It often provides the researchers with ideas,
approaches, and clues they may not have foreseen
before conducting the pilot study. Such ideas and
clues increase the chances of getting clearer
findings in the main study (G.A. Lancester, et al.,
2004; Thabane. L., 2010).6

7. Questions for Pilot Study
1. Process: This assesses the feasibility of the steps
that need to take place as part of the main study;
2. Resources: This deals with assessing time and
budget problems that can occur during the main
study;
3. Management: This covers potential human and
data optimization problems such as personnel and
data management issues at participating centers;
4. Scientific: This deals with the assessment of
treatment safety, determination of dose levels and
response or other interventions, and estimation of
treatment effect and its variance (Van Teijlingen. et
al., 2001; Thabane. L, et al., 2010)
7

8. Questions for Pilot Study (2)
G.A. Lancester, et al., 2004; Thabane. L., 2010
Process: This assesses the feasibility of the processes
that are key to the success of the main study
 Sample size and sample characteristics for the main study
 Is it obvious who meets and who does not meet the eligibility requirements?
 Are the eligibility criteria sufficient or too restrictive?
 Recruitment rates
 Clinical or surrogate end-point for the main study
Resources: This deals with assessing time and
resource problems that can occur during the main
study
 Retention rates
 (Non) compliance or adherence rates
 Understanding of study questionnaires or data collection tools:
 Do subjects provide no answer, multiple answers, qualified answers, or
unanticipated answers to study questions?
 Length of time to fill out all the study forms
8

9. Questions for Pilot Study (3)
G.A. Lancester, et al., 2004; Thabane. L., 2010
Management: This covers potential human and data
management problems
 Determining capacity:
 Will the study participants overload your phone lines, mail box, or overflow
your waiting room?
 Determining process time
 How much time does it take to mail out a thousand surveys?
 Are the equipment, computer, and system readily available when and
where it is needed?
 What happens when it breaks down or gets stolen?
 Can the software used for capturing data read and understand the
data? What’s its capacity?
 Determining center willingness and capacity
 Do the centers do what they committed to doing?
 Do investigators have the time to perform the tasks they committed to
doing?
 Are there any capacity issues at each participating center?
9

10. Questions for Pilot Study (4)
G.A. Lancester, et al., 2004; Thabane. L., 2010
Scientific: This deals with the assessment of
intervention, treatment safety, dose, response, effect
and variance of the effect
 Are the interventions accepted?
 What challenges do study personnel have?
 Are there any problems entering data into the computer?
 Can data coming from different sources or activities be matched?
 Is it safe to use the study drug/intervention?
 What is the safe dose level?
 Do patients respond to the drug?
 What is the estimate of the treatment effect?
 What is the estimate of the variance of the treatment effect?
10

11. Sample Size
G.A. Lancester, et al., 2004; Thabane. L., 2010
 In general, sample size calculation may not be
required for some pilot studies.
 It should be representative of the target study
population.
 It should be based on the same inclusion/exclusion
criteria as the main study.
 It should be large enough to provide useful
information about the aspect that are being
assessed for feasibility.
11

12. Data Analysis
G.A. Lancester, et al., 2004; Thabane. L., 2010
 Conclusion as a target:
 Stop  main study not feasible
 Continue but modify protocol  feasible with
modifications
 Continue without modification but monitor closely 
feasible with close monitoring
 Continue without modification  feasible as is.
 The analysis of a pilot study should be mainly
descriptive or should focus on confidence interval
estimation; avoid statistical significance, hypothesis
testing, or powered test.
12

13. Reporting
G.A. Lancester, et al., 2004; Thabane. L., 2010
 The title or abstract should indicate that the study is a “pilot” or
“feasibility”
 Clearly define why do we need a pilot and relate with the
specific scientific primary and secondary objectives and
hypotheses for the main study; Clearly define what are the
outcome measures of the pilot; It is important to clearly
indicate the feasibility objectives as the primary focus for the
pilot.
 Interpretation of the results should focus on feasibility, taking
into account the stated criteria for success of feasibility, study
hypotheses, sources of potential bias, confounding factors,
and the risk associated the study.
 Results from hypothesis testing should be treated as
preliminary and interpreted with caution, as no formal power
calculations have been carried out.
 Discuss the external validity  as default, the result may not
generalize to the target population.
13