Sami Grimes and Heather Treizenberg of the National Sea Grant Office discuss national reporting guidelines and response to network feedback. Sea Grant Week 2010
2. Outline
• Components of an annual report
• How annual report information is used
• How do annual reports fit into the planning,
implementation and evaluation (PIE) system
• Network concerns
• Feedback Results
• What to expect for next year’s annual reporting
process
• Discussion/Questions and Answers
3. What are the components of
an annual report?
• Program and Project Impacts
• Performance Measures
• Program Metrics
• Project Updates
Dates – follow grant year (2/1 – 1/31)
(3) Perf.
Measures
(2) Program
Metrics
(1) Program &
Project Impacts
(4) Project
Updates
Dates follow grant year (2/1 – 1/31)
4. Sea Grant Milestones from
2009-2010 Annual Report
4
3,500 jobs and 650 businesses created or retained
SG assisted 160 coastal communities adopt or
implement hazard resiliency practices
1,700 undergraduate and graduate students
supported
128 peer-reviewed journal articles published
Leveraged over $86M
186 coastal communities restored degraded
ecosystems (nearly 32,000 acres)
Programs now have access to the national information from the National website.
5. How is annual report
information used?
Nationally -
• Mandatory reporting to NOAA, Dept. of
Commerce and Office of Management &
Budget
• Budget request narratives
• Tell the national and state
program stories
• Create Hill briefing materials
6. How is annual report
information used? (con’t)
• Assess progress towards program plan
(annual self-evaluation)
• Create Performance Progress Reports for
NOAA grants online requirement
• Other sea grant program reporting or
communication purposes
7. How do annual reports fit
into the PIE system?
• Track and report progress against plans over
a one-year period (program self-evaluation)
• The current program annual report goes to
members of the Site Review Teams
• Basis for discussion of the program for the
NSGO Review (information sharing)
• Beginning in 2015, a collection of annual
reports (2010-2014) will serve as the basis
for the Performance Review Panel review
8. Annual Report - Network
Concerns
To better serve the network: we listened,
we implemented:
• Weekly annual report conference calls
• Impact Guidance
• Definitions for Metrics and Perf.
Measures
• Invited feedback
9. Methods
• SGA & Advisory Board
• Census of programs (via directors)
• Online feedback form (surveymonkey.com)
• September 17 – October 1.
– 1 invitation, 1 reminder/thank-you
– No non-response check
• 18/32 programs completed form (56%)
13. Results
• Comments on performance measures:
• ”Easy to count attendees of workshops; but more difficult to
determine who modified practices. Difficult to determine who
implemented practices and how much SG was a factor.”
• “The definitions, and deciding exactly how to count what things,
remains a challenge. For example, it is quite subjective and
variable to define (a) a coastal community, (b) a sustainable
practice, and (c) whether a practice has been adopted or
implemented specifically as a result of our program, thus making it
difficult to accurately measure that particular performance
measure.”
20. Results
• Please describe ideas for how the National Sea Grant
College Program can improve planning and reporting
processes that best capture the excellence of Sea Grant
Programs. (Please type below.)
– Align impacts and performance measures reporting with grants
reporting.
– Appears to be little relationship between omnibus and PIE.
– Administrative burdens have become an enormous drain.
– We worry about too much reliance on data output without
analysis or thoughtful synthesis.
– Increase capacity of National Office.
– We need to get beyond the numbers to the qualitative impacts
on people/communities wherever possible.
21. Results
• General comments:
– Provide feedback to programs on how data were used.
– We appreciate the efforts that the NSGO staff are putting in
trying to shift the network into this new system. It's been as
difficult for the National Office as it has for the programs. We
also recognize that NOAA, OMB and Congress can always add
new expectations. We're heading in the right direction, and if we
can keep to the path we are currently following for the
foreseeable future with few changes, it will benefit the entire
network. Thanks!
22. What to expect for next year’s
annual reporting process
• Guidance will be sent out by Feb. 2011
• Content of annual report will be similar to
last year, but will also include progress
towards your program plan
• Mechanism to collect the information is
currently unknown
• NIMS – programs have an opportunity to
help shape the system – become a beta-
tester!
- What exactly IS an annual report
- How annual report information is used
Network concerns
- How Annual Reports fit into the planning, implementation and evaluation system
Results from the Feedback Questionnaire that took place in Sept.
- What to expect for next year
- Then we will have plenty of time for discussion and Q&As
Four main components to the annual report:
Program & Project Impacts,
Program Metrics,
Performance Measures, and
Project updates
The 1st 3 components are what are produced as a program’s annual report for Site Visits.
Dates of the annual report follow the grant year (2/1 – 1/31)
The annual report is intended to highlight annual components of the program
Same information is collected from each program in order to aggregate impacts, performance measures and metrics up at the national level to help highlight components of the entire national sea grant college program and help tell part of the national sea grant story
Because we collect similar information from the program, we are able to help tell a better national story.
Last year alone…the sea grant network “created or retained...”
read bullets in slide
Information from bullet 2, 3, and 4 has already been used and requested by NOAA to report to DOC and OMB
These types of bullets help us tell part of the national story; the other part is the impact statements
Mandatory Reporting:
Reporting – quarterly, data calls/taskers – weekly, hot topics
Firedrills – give us 2-24 hours
Budget Request Narratives
Template to follow consists of half of the allowed 1page to consist of metrics, milestones and PM
Those that are making budget decisions focus on the budget amount requested and then look at the table that includes the metrics, pm and milestones without ever reading the narrative – looking at the quantitative data, and little opportunity for the qualitative impact stories
Providing information is essential to survival – some of the exercises aren’t about budget increases, but maintaining our current budget
And then, of course, telling the national story or state program story, depending on the tasker
Other ways annual reports are used -
As part of the new evaluation system, annual reports are to serve the purpose of a self-evaluation; programs can annually review their progress towards their program plan and make any necessary adjustments in order to meet their planned outcomes
Programs can use the information in the annual report to create their performance progress report
Nearly 80% of the feedback questionnaire respondents said they used the annual report content for other reporting or communication purposes (more about the questionnaire results to come soon)
As stated in the previous slide, annual reports track and report progress against program plans over a one-year period (program self-evaluation)
The most recent annual report goes to members of the Site Review Teams – give them an idea of the activities the program is involved in
Annual reports serve as a basis for discussion of the program for the NSGO Review (information sharing)
Beginning in 2015, a collection of annual reports (2010-2014) will serve as the basis for the Performance Review Panel review
Rumblings in the network over Annual Report Process – developed feedback questionnaire to unearth and understand what the issues and concerns were
Majority of respondents felt they provided input, but not enough time was given and timing was poor.
Comments:
Execution of annual reporting process has been poor, the idea is important.
It is a lot of work to put this together; not sure if the equivalent benefit has been realized.
Programs need an explanation of the annual reporting and how this information has been used.
In university setting, programs are aware 1 year out of the reporting requirements not post-year.
Not clear how annual reporting information will be used in PIE.
A database can’t replace the human thought needed to synthesize, aggregate, and analyze the impact information.
Comments:
Once guidance was finalized, it was great, but along the way staff lost significant time awaiting its announcement.
The earlier the better, ideally when the reporting year starts (Feb 1). At least 8 weeks are needed to provide information. “We need final guidance by the end of the year for annual reports due next year.”
“Defensible activity” only counts relative to the definitions provided.
Website is not easy to find information.
Deepwater horizon incident in April changed timelines.
University reporting is due before NSGO annual reporting, so programs already go started on the tracking/reporting even before final NSGO guidance was issued.
Problem of multiple changing deadlines – lots of scrambling and wasted time.
Comments:
Agreement that national performance measures do not capture national or program stories.
Comments:
Comments:
All online resources MUST be Mac-compatible!!!!!
Comments:
We use info to report to university, Board of Governors, UPR, College Extension reporting system, advisory bodies, reporting progress reports to NOAA, SGA, website, constituents via magazines.
We do not yet trust NIMS, but staff generate other reports.
Program re-purposed impact statements for university reports.
“The annual report is a secondary use for data collected for state-level reporting.”
Comments:
To university and extension administrators, as well as board of governors.
Faculty members report to individual departments.
Other funding sources require periodic reports.
Director’s performance reviews.
Comments:
“Yes, but the NSGO reporting requirements are increasingly diverging from what we report to our Board and adding to our reporting load.”
“There are different formats, expectations, "years" and timing. University reports are due in May/June, but we utilize the material we are pulling together for NSGO and other grant reports to put those reports together.”
Guidance will be sent out by Feb. 2011
Information for annual reports will be the same as last year, but include progress towards your program plan
Mechanism to collect the information is currently unknown
NIMS – programs have an opportunity to help shape the system – become a beta-tester!
KOLA – WHERE ARE WE WITH UPDATING NIMS???