Guiding Government Investments in Research Using Data-Driven Scientific Program and Policy Development

Gap Analysis: Guiding Government
Investments In Research Using Data Driven
Scientific Program and Policy Development

Norman S. Braveman, Ph.D.
President
Braveman BioMed Consultants

Elsevier Reed Conference
Accelerate Research Excellence: Evaluate, Benchmark and
Strategize
September 23, 2001
New Delhi, India

Braveman BioMed Consultants - "Working with you for excellence in research." 1

 Braveman, N.S. What science indicators can tell policymakers. In Margaret O.
Meredith, Stephen D. Nelson and Albert H. Teich (Eds.) Science and Technology
Yearbook: 1991. Washington, D.C.: American Association for the Advancement of
Science, 267-278.
 Braveman, N.S. Evaluation at NIH: The policy issues. In Susan L. Sauer (Ed.) R & D
in FY 1989: Looking Ahead in an Election Year. Washington, D.C.: American
Association for the Advancement of Science, 1988.“Evaluation at NIH: The policy
issues”

 Policies based on data are better than those that are not
 The primary driver or source of data for development of
science policy and programs should be science itself
 But we need to be cognizant of other drivers and sources of
information
 AS A GRANT MAKER YOU HAVE AN EXTREMELY IMPORTANT
IMPACT ON SCIENCE

2
© Braveman BioMed Consultants - "Working with you for excellence in research."

 So what prevents decision making based on science from
happening?

 Limited and/or uninformative data –> need for additional and
new metrics (demographic/bibliometric vs. content)
 Lack of access to or difficulty in accessing data –> getting data
is not easy and time consuming for first level users
 Austere times –> tendency to not develop
 Prevailing attitudes about formal planning in science

3

Prevailing attitudes toward PP&D in science

 It‟s not possible or advisable to plan science -> can lock you into
unproductive pathways

 It‟s important to be flexible and able to change directions quickly
to follow leads/new directions

 „Peer review‟/publication of findings are the only endpoints that
matter in determining future directions

 Serendipity or chance historically lead to great findings –> the
double helix; clinical applications of iPS

© Braveman BioMed Consultants - "Working with you for excellence in research." 4

"In the fields of observation, chance favors only the prepared mind."
Louis Pasteur

A „prepared mind‟ (in science) is one that has considered
alternative paths, even in an informal way, particularly when
results don‟t turn out as expected.

“A bad is experiment is one that will provide answers only if the
hypothesis is supported or disproven. A good experiment is one
that will provide answers no matter what the outcome of the
experiment.”
Anonymous Peer Reviewer


 The context
 Some sources of support for research in the US
 Current interests of grant makers

 An approach to Data Driven Scientific Program and Policy
Development using PP&D
 Logic Analysis
 Strategic and Implementation Planning
 Gap Analysis

 Scientific fingerprinting – a way to collect data for PP&D of science
policies and programs
 What is scientific fingerprinting
 Examples of current use in PP&D

6

2011 Agency Budget Request
Funding Agency (% for R&D)
National Institutes of Health $32.1 billion (85)
National Science Foundation $7.4 billion (80)
Department of Energy $28.4 billion (18)
NASA $19.0 billion (33)
Department of Defense $708.0 billion (10)
US Department of Agriculture $146.0 billion (2)
Various State Funding Programs
(stem cells, tobacco –related --
diseases)

7

 There are approximately 4000 private grant makers
in the US

 Health Research Alliance:
 not-for-profit, non-governmental funders of biomedical
research and training from biomedical science to
applications that advance health
 47 member organizations
 HRA member organizations provide over $1.5 billion in
research awards each year, funding over 5500 researchers
annually

8

Many of these sources of funding relies on
PP&D in order to

stay current in their area of science
be ahead of the curve in planning for future
scientific developments
Insure that their resources are used appropriately
and in the most effective way
possible…particularly in austere times

9

 Scientifically meritorious research
 Innovative research that will move the field ahead
 Development of the research workforce of the future
 Basic scientists
 Clinician and clinical scientists
 Interdisciplinary scientists
 Diversity in racial and ethnic backgrounds
 Topic of the day
 Personalized medicine
 Translational research
 Systems approach to biomedicine
 Stem cell research
 Disease of the day
 Obesity
 Cancer
 Heart disease

10

The foundation of PP&D is evaluation…

a systematic method for collecting, analyzing, and using data
and information to answer questions about projects, policies
and scientific programs

„Data „ is not the plural of „anecdote‟. Unknown


 Demographic/Quantitative
Who is receiving funding
When did they receive it
Where are they located
How much did they receive; how much in total is
being spent on the area of science
How many publications/inventions have resulted
from the research (return on investment)
How many publications have resulted from grants


 Demographic/Categorical

 Content Based/Qualitative
 Captures the content of the research in the portfolio ->
What is the content of the research that is being addressed?
What isn‟t being addressed…and why?
 Systematic, comprehensive, reliable and valid
 Can link a current portfolio to portfolios that exist in other
organizations (e.g., funding agencies)
 Can add important scientific information in addressing
some of the quantitative questions



What are the goals of PP&D for organizations that
fund research?

 To maximize the impact of funding by advancing science
 To determine whether what has been done has met the
specific goals and objectives of the organization
 To chart the future priorities and directions of the
organization


Logic Model
 Clear statement of mission and goals of the organization
 Identification of inputs and outputs
 Identification of expectations and measures of success

Strategic Planning
 Statement of values, mission, priorities and goals

Implementation Planning
 Timelines, benchmarks, mechanisms, sequence of events

Gap Analysis


 A Major Component of PP&D

 Answers the questions –
Where are we?
 Where do we want to be?
 Are we making the best use of our resources?

“It‟s the difference between what an organization is
doing or has been doing and what the
organization should be doing or wants to do…”
Source Unknown


Applied from a business/economics environment to organizations
involved in funding research

 Identifies the gap between the optimized allocation and
integration of the inputs, and the current or future level of
allocation

 What are the inputs in the world of science? -> money, people,
facilities/equipment, knowledge

 What do we mean by “optimization” or “integration ” when we‟re talking
about funding of science? -> a unique combination of inputs that
yields best performance on outcome measures


Applied from a business/economics environment to organizations
involved in funding research
 Identifies the gap between the optimized allocation and
integration of the inputs, and the current or future level of
allocation

 Used to compare a program‟s actual performance with its
potential performance

 How do we determine a funding organizations potential performance?
 from strategic plan, implementation plan and logic model

 How do we measure a funding organization‟s actual performance?
 use outcome measures developed from logic model


Also addresses specific questions about the current
state-of-the-science as well as about pathways to
scientific advancement
What do we need to do to move ahead?
What are the barriers and where are the
opportunities?

Gaps are then defined in terms of the current
portfolio of the organization and the near-
and long-term state-of-the-science


Strategic Plan
Background & Outcome
Stakeholders Measures
Goals

Logic Model

Gap
Analysis
Expert Panel
Current
•Future of area portfolio vs. Data
•Barriers “ideal” Collection
•Possible priorities portfolio

Organization Data-Driven Decision Process


 Background (stakeholder input; source documents)
 What is the organization‟s role in the big picture?
 What are our present goals?
 Where are we going?
 How have we gotten to where we are?
 What has changed that we need an updated assessment?
 Who are the other participants and what is their role?
 What is/has been our relationship with them?


 Background

 Portfolio Analysis (data collection)
 What has the organization accomplished toward current
goals?
 What science have we been funding? What science have we not
been funding?
 What is/are the scientific content of the same or a similar
portfolio(s) of other organizations?
 What science areas are covered by applications that have been
submitted to our organization?
 Of those submitted to our organization, which ones are being
funded? Which ones are not being funded…and why?


 Background
 Data collection
 Gap Analysis
 Compare content of current portfolio with „ideal‟ portfolio
 Assess content of current support by all organizations for
duplication, overlap and collaboration
 Compare the content of current portfolio(s ) with the future of
science
 Compare content of all submitted applications with program goals
and future of science
 Compare content of successful applications with program goals
and future of science
 Compare content of unsuccessful applications with the future of
science and program goals


 Background
 Data collection
 Gap Analysis

 Expert Panel (thought leaders)
 Where is the field going in the short-/long-term?
 What are the scientific barriers to getting there?
 Assessment of current content of programs by all organizations
for duplication, overlap and collaboration
 Where are the gaps between the content of current portfolios and
the future of science…with „ideal‟ portfolio
 Recommendations/priorities about where the organization being
analyzed can make significant and unique contributions/impacts


These are the drivers of the
development of science policies and
funding programs

But they aren‟t the only ones…the
final step -> organizational decision
making


 Background
 Data collection
 Gap Analysis
 Expert Panel
 Organizational Decision Making (stakeholders and
leadership)
 What unique scientific contributions do we want to be
known for?
 Resources available – people, fiscal
 Possibilities for collaboration/leveraging with other
organizations
 Political considerations
 Determine which gaps are going to be addressed
 Develop timelines and benchmarks


 A representation of a document generated with the
Elsevier Fingerprint Engine

 A precise abstract representation of a text which
allows us to “look into the text” (content) rather
than only looking at the meta data (demographics).


 NIH Research, Condition, and Disease
Categorization Process (RCDC)
 Introduced in 2009
 A computerized process used at the end of each fiscal year
to sort and report the amount funded in each of 229
historically reported categories of disease, condition, or
research area.
 The Florida Biomedical Research Program –
scientific program management and development
 National Heart, Lung and Blood Institute (NIH) –
identifying ideas and tracking progress


Florida Biomedical Research
Program:
Integration of PP&D Within
An Ongoing Scientific
Program


Logic Model Strategic Plan
Portfolio Analysis • Program Goals
• Outcome
Program Planning

Measures Implementation Plan
Gap Analysis
Recommendations
Expert Science Panel

Council Review
Program Development

Call Development Logic Model & Gap Analysis
• Priority science areas • Program Goals & Objectives
• Funding mechanisms • Outcome Measures
• Application configuration • Gap Analysis
Biomed Experts &
Reviewer Finder
Issue RFA(s)
Grantsmanship Seminars
(Answer questions about RFA)

Applications
Applications

Summary Statements
Grants Management & Peer Review
To Applicants
Reviewer Finder Systems

Council
Review & Funding Notice of Award
Recommendation
Grant Monitoring

Progress Reports
Grants Management &
Annual Report
Program Review
•Grant Outcomes
Financial Reports •Success Stories
•Program Outcomes


National Heart, Lung and Blood
Institute (NIH)

Identifying ideas and tracking
their implementation through
program development

SCIENTIFIC
WORKSHOPS I
PROFESSIONAL
C NATIONAL
GROUPS
L ADVISORY
DHHS LEADERSHIP
E
G
COUNCIL
NIH OFFICES I OR BOARD
S IC Staff
L
Ad Hoc Advisory
Groups A

PATIENT
Opportunity T
E Decision
GROUPS D Process
CONGRESS M
I
OTHER FEDERAL S IC NIH Early
AGENCIES S ADJUSTMENTS Notification
I
EXTRAMURAL O
System
SCIENTISTS N
INDUSTRY/
PRIVATE SECTOR
ISSUE RFA OR PA


36



37

38

The effective use of data driven PP&D in developing
science policy and funding programs is dependent
on metrics that reflect the scientific content of the
policy or programs as well as demographic
information.

The metrics must be:
 reliable, valid, and meaningful to the user
 accessible, interpretable, and readily analyzable by the user

And the system used to compile and summarize the metrics
must be adaptable to a variety of circumstances


Guiding Government Investments in Research Using Data-Driven Scientific Program and Policy Development

Recomendados

Recomendados

Mais conteúdo relacionado

Mais procurados

Mais procurados (20)

Semelhante a Guiding Government Investments in Research Using Data-Driven Scientific Program and Policy Development

Semelhante a Guiding Government Investments in Research Using Data-Driven Scientific Program and Policy Development (20)

Último

Último (20)

Guiding Government Investments in Research Using Data-Driven Scientific Program and Policy Development