This video is meant for Extension educators to demonstrate the feasibility and usefulness of an Extension program and Evaluation Strategy that is based on specific goals. This presentation is a basic version and we have much more information that is part of continuous improvement in the slideshow. We will share other presentations with more information - so this is just the beginning! For program evaluation/monitoring questions, call 251-331-8416 or email bugdoctor@auburn.edu. For looking at some of my IPM Program evaluation publications, visit www.aces.edu/go/87 and click on 'IPM Evaluation Toolkit' in the menu. Thank you.

1. How to Improve Extension
Program Outcomes using a
Monitoring and Evaluation
Strategy
Dr. Ayanava Majumdar
Extension Entomologist, SARE Coord. (AU)
111-A Duncan Hall, Auburn University
bugdoctor@auburn.edu
251-331-8416

2. Focus of this presentation
• Structure & implementation of Alabama IPM
Program
• Provide some basics of evaluation – Extension
perspective!
• Encourage you to be consistent in your
evaluation efforts
• Develop capacity building within organization

3. About the Author (Ayanava)
Ext. evaluations:
• Memberships: American Eval. Assoc., Southeast Eval. Assoc.
• AEA365 Blog Curator (2010-2011), Organization Capacity
Building TIG Grant Panel, Ext. Eval. TIG member
• Lead evaluator: USDA-NIFA, AL DoA, commodity grants,
School IPM, AACAAS, NACAA
Organizational capacity building:
• Initiated ACES Program Evaluation Resource Committee,
2010… created the Evaluation Toolkit [online]
• Workshops, webinars, publications: total ~7 hr, trained 161
Agents, County Coordinators, Specialists, Ext. Administrators

4. Alabama Small Farms
• Produce fruits & vegetables that
are consumed locally
• Area = 6,000+ acres
• Crop value = $48 million
• Growth rate for industry = 14%
• Vegetables: Tomatoes, sweet
corn, watermelon, crucifers
• Farmer markets: 135 (1100 farms)
• Direct sales from AL farmers to
consumers account for 0.2% of
farm sales (Meter 2012)

5. Alabama Cooperative Extension System
Ref.: Robinson, Dubois, Bailey (2005). Journal of Extension.
Regional Extension Agents (REAs)
Regional or statewide IPM
events
Outputs change annually
Outcomes do not change!
Impacts now underway.
+ CECs

6. ACES Commercial Horticulture Team –
Vegetable IPM Team Members

7. ACES Home Grounds Team – Vegetable IPM
Team Members
Chris Becker, REA Willie Datcher, REA Mike McQueen, REA
Alfred Jackson, Tuskegee Extension

8. Awards & Recognition
• ‘Friends of IPM - Pulling Together Award’ from the Southern
Region IPM Center, Raleigh, NC (2014) for IPM newsletter.
• Friends of IPM: Future Leader Award (2012)
By Southern Region IPM Center, Raleigh, NC
Recognizes excellence in IPM program directly
• Search for Excellence – Crops (2012)
By National Association of County Agricultural Agents
Recognizes a high impact IPM program
• Achievement Award (2012)
By the National Association of County Agricultural Agents
• Communication Awards (6)
By National Association of County Agricultural Agents

9. Structure of Alabama IPM Program:
A Model Program for Study

10. Fund acquisition (external)
• Funds needed to support REA travel & IPM
demonstration plots
• USDA/NIFA Collaborative Grants: $124,500 (part
of two large grants worth $2.1 million) = 6%
• SARE support: $10,000 + personnel for website
• Alabama Department of Agriculture: $25,000
• Industry support: $45,000

11. Theory of Planned Behavior (TpB)
Ajzen, I. 1991. The Theory of Planned Behavior. Organizational Behavior and Human
Decision Processes. 50: 179-211

12. Theory of Change: Ext. Logic Model

13. IPM Theory of
Change
Communities
ImpactsModified from Kokate et al. 2009
Farmer-to-farmer
Train-the-trainer
Sustain change
Key farmers
Technology acceptance
ExtensionResearch
Project Management
Capacity Building
Need identified
(context)

14. Ext. IPM Project Implementation
Continuous needs
& outcomes
assessments
1
2
3
4
1
2
3
4
Model by Dr. A. Majumdar 2011
Based on Rockwell et al. 2003

15. CommunicationStrategy(IPM-CORE)
No cost tools
Technology identified & assessed
Story developed
Press release
Extension Websites Social sharing
(Slideshare, YouTube)
Social networking
(Facebook)
Medium cost tools
IPM Communicator
Newsletter
Extension regional
meetings, IPM
Web Conferences
IPM Project
flyers,
bookmarks
High cost tools
Multiple partnerships (~20)
IPM Exhibitions,
Newspaper
advertisements
IPM on-farm
demonstrations/
communications
Small Farm
IPM Field
Guide
IPM App?

16. Level 1: Regional IPM Meetings
• Participants: Vegetable producers, certified
organic farms, gardeners & homeowners, crop
advisors, nonprofit organizations (partners)
• Teaching techniques: Presentation, pest
specimens
• Typical length of talks: 46 min
• Participants total: 1,535
• Number of events completed: 58
• Evaluation: Events are monitored for quality
and impacts via multimodal surveys

17. Regional IPM Training Meetings (Level 1 training)

18. Level 2: IPM Workshops
• Participants: Small vegetable producers, low
resource farmers, Master Gardeners, nonprofit
organizations, Extension Agents (in-service)
• Teaching techniques: Case studies, simulations,
specimens
• Typical length of talks: 140 min
• Participants total: 570
• Number of events completed: 14
• Evaluation: Qualitative & quantitative feedback,
impact evaluations

19. Hands-on IPM Workshops (Level 2 training)

20. Level 3: IPM Field Days
(Demonstration Sites)
• Participants: Producers, low resource farmers,
gardeners
• Teaching techniques: Scouting, pest
identification, decision-making system
• Typical training time: 3-4 hours
• Participants total: 308
• Number of events completed: 12
• Evaluation: Qualitative & quantitative feedback,
impact evaluations, farm visits

21. IPM Field Days (Level 3 training)

22. Farmer-to-farmer training model is
very influential for IPM project

23. Farmer-to-Farmer training
during IPM Field Days
Causes high information retention &
impact on communities

24. Basics of Program Monitoring &
Evaluation (PME)

25. Evaluation
• Evaluation is the systematic collection of information
about the merit, worth or significance of a program
(Scriven 1999).
• Evaluation is broadly defines as the systematic
collection of data – both quantitative and qualitative
– to aid users in developing knowledge about &
managing a targeted set of activities (Scheirer 2012).
• Ext. evaluation practice aims to find plausible
conclusions, not cause & effect!

26. Rate your IPM program evaluation experience
(useful exercise for audience)

27. Common myths!
• My clients like me, so I do not
need to evaluate.
• Evaluation is hard & outside my
job description.
• Public does not respond to my
surveys.
• Nobody is going to look at the
evaluations once they are done.
• Only ‘experts’ can do evaluation.
Why bother!
PERCEPTION REALITY
• Evaluation helps document your
success.
• 20% effort generates 80%
information.
• Provide time to respond, part of
agenda, show improvements!
• Share the results with all
stakeholders.
• Evaluation can be done by any
trained individual committed to
the standards.

28. Who can do the evaluation?
Internal evaluators
• Project leadership team
directly guides evaluation
(cost-effective)
• Indicates organizational
commitment /capacity
building
• Project monitoring possible
(utilization-focus)
• Fast assessment of reactive
programs
• Good participation
External evaluators
• Neutral third-party
observers (expensive)
• No internal capacity
building
• Good for impacts
• Slow assessments & less
useful
• Poor participation (trust)
Based on Patton (1997)

29. CDC Framework for Program Evaluation Standards
Thomas Chapel, Chief Eval. Officer, CDC

30. Purpose of Evaluation
Formative evaluation
• Done when the program
is active
• Project monitoring
• Continuous use
• Flexible (mixed) designs
Summative evaluation
• Done at the end of a
program
• Measure impacts
• One time use
• Rigid design, expensive
Patton (1997), Scriven (1972)

31. Utilization-focused Evaluations
• Evaluation technique emphasizing USE (Patton 1993)
• Improvement-oriented evaluation: includes formative
evaluations, quality enhancement, Total Quality
Management (TQM)
• TQM includes using information systems to monitor
program efforts & outcomes continuously
• Use feedback as a monitoring system (like insect traps!).
+ Trouble= =
Do something
about it!

32. Evaluation techniques
Reactive assessment:
• Participants are aware of the
assessment
• Examples: surveys, interviews,
tests
Non-reactive assessment:
• Participants are unaware of the
assessment
• Examples: observation
Haas (2005)

33. Choose outcome ‘indicators’ carefully
• Indicators are critical questions
• Based on program objectives
• Used for monitoring progress
• Usually expressed as rates, percentages,
efficiencies, etc.
• There should be few trackable indicators
(QUAL and QUAN)

34. Taylor-Powell & Henert 2008

35. Needs
assessments
Process
evaluations
(formative)
Outcome
Evaluations
(formative)
Impact
evaluations
(summative)
Project
monitoring
Project
improvement
Modified
strategy
Modified
outputs
Evaluation Strategy in Extension IPM Program
Graphic by Dr. Ayanava Majumdar, ACES

36. What do we measure?
Hierarchy of Outcome Effects
• Level 1: REACTION
• Level 2: LEARNING
• Level 3: BEHAVIOR
• Level 4: RESULT
Increasing
complexity of
evaluation
Evaluate
during
program
Kirkpatrick Four Levels, 1959, 1994
Post
program
evaluation
QUAN
QUAL

37. Hierarchy of Effects in IPM Program
(In Pictures)
• Level 1: REACTION
• Level 2: LEARNING

38. Hierarchy of Effects in IPM Program
(In Pictures)
• Level 3: BEHAVIOR
• Level 4: RESULTS

39. AL IPM Project
OUTCOMES
(Usefulness of Project
Monitoring)

40. Project outcomes (2008-2012)
• Developed a unique Theory of Change and
Transformational Extension Education Model based
on long-term outcomes
• New producers reached 14% and rising (transitioning
farms 7-10%).
• General IPM adoption rate: over 90%
• Adoption rate for unique IPM tactics: 63% (up from
38%)
• Publication in print & electronic have significantly
reduced barrier to IPM adoption
• Funding for Small Farm IPM program has doubled
since 2012. Much focus on Ext. Agent training.

41. • Rising interest of producers/Pull on project:
– Over 10 times rise in participation in IPM events
– 10 times rise in number of IPM events
• Barriers to IPM adoption (crop advisors):
– Lack of awareness reduced by 33% in 3 years
– Difficulty in accessing information reduced by 33%
• Barriers to IPM adoption (producers):
– Lack of awareness reduced 16% in 3 years
– Difficulty in accessing IPM information reduced 13%
Project outcomes (2008-2012)

42. IPM Theory of
Change
Communities
ImpactsModified from Kokate et al. 2009
Farmer-to-farmer
Train-the-trainer
Sustain change
Key farmers
Technology acceptance
ExtensionResearch
Project Management
Capacity Building
Need identified
(context)

43. 67
259
505
945
39
54
57 55
3 4 28 270
100
200
300
400
500
600
700
800
900
1000
2009 2010 2011 2012
Total participants
Survey return rate (%)
Major IPM Events
GROWING INTEREST OF FARMERS IN IPM

44. 1.5
6 6.9
73
36
26
5
8 10.18
34
29.1
11
7
20.9
0
10
20
30
40
50
60
70
80
2009 2010 2011
Beginning farmers (%)
Conventional producers (%)
LRFs (%)
Transitioning farms (%)
Gardeners (%)
NATURE OF AUDIENCE AT IPM EVENTS

45. 28
13.6
9.3
0
5
10
15
20
25
30
2009 2010 2011
Average land holding (acres)
CHANGING NATURE OF EXTENSION CLIENTELE
(reason for splitting IPM program into two campaigns to
increase impacts)
Organic/
Small Farms IPM
Campaign
Conventional
farmers
(with 15 to 50
acres)
2012
Farmers
(<10 acres)
Gardeners
(<2 acres)
Conventional/
Large Farms IPM
Campaign

46. Conventional vegetable farmers = 15-50 acres
Transitioning farmers = 10-14 acres
Beginning farmers = <10 acres
Low resource farmers = 2 to 4 acres
CHANGING NATURE OF EXTENSION CLIENTELE
(reason for splitting IPM program into two campaigns to
increase impacts)
Two IPM Campaigns launched in 2012 with separate
strategies and outcomes. Rapid increase in small
farm IPM funding and number of participants due to
rising interest in local food systems.

47. 45
29
10
23.2
30
2.2
20
11.4
2.5
5
0
5
10
15
20
25
30
35
40
45
50
2008 2011
Lack of awareness (%)
High cost (%)
Difficult to access information (%)
Low availability of insecticides (%)
Lack of time (%)
BARRIERS TO IPM ADOPTION & CORRECTIONS
Major concerns at the time

48. PROJECT IMPACTS
(Watch 2013 Impact Video at
http://youtu.be/aqrjQINLUdw)

49. Impact Evaluation, 2010-2011 (n=58):
• E-subscriptions growth rate = 12%
• Current subscription base ~1350 with much higher readership.
• 34% farmers, 28% company representatives, 38% others
• 53% read for 15 minutes and 22% for 30+ minutes
• 94% support continuation of the newsletter (2011 survey)
• 70% use the information (2013 survey)
• Six financial gains reports (2013): $3,550 in pesticide saving
• Short-term impact of publication: $591 per case of adoption
IPM Newsletter Impacts

50. Program Impacts: IPM Adoption
Indicator 2011 2010 2009 Increase
Adoption of insect
monitoring/scouting
practices
73.5% 41.0% +32%
Adoption of insecticide
recommendations
79.6% 46.5% 33%
Use of biological
insecticides
42.3% 26.5% 16%
Overall IPM adoption 62.7% 38.5% 24%
Economic & social impact assessments
ongoing in 2012 and beyond.
Received IRB Approval for Surveys.

51. Economic impacts
• Yield loss in the absence of IPM: 44% or
more
• Vegetable producers gained $246 per acre by
using IPM recommendations
• Profits occur by judicious use of pesticides,
adoption of appropriate control tactics after
pest identification, and timely action.
• Weekly IPM newsletter resulted in saving of
over $500 per adoptive farmer

52. • Increased use of biological insecticides
(42%)
• Conservation of natural enemies
(unknown)
• Profits occur by judicious use of pesticides,
adoption of appropriate control tactics
after pest identification, and timely action.
Environmental impacts

53. • Three new partnerships with influential
commodity & consumer organizations
• New partnerships with nonprofit
agencies serving LRFs in Black Belt of
Alabama
• Impact assessments continue in 2012 &
beyond.
Social impacts

54. Charles Brannon, vegetable producer (Addison, AL): “This is the first year we used
the IPM recommendation from the handbook and received training from Dr. A. In
the hoop house alone, we sold about $5,500 worth of tomatoes which is double the
output from last year. Fruitworms and stinkbugs used to do about 50% yield loss
but not this year. With abundant production, we are taking our produce to large
farmer markets in Birmingham and Decatur, not limiting ourselves just to the farm
stand.” (Surveyed on July 5, 2012)

55. Danny Dickie, vegetable producer (Oneonta, AL): “We use the
insecticide recommendations from the SE Vegetable Handbook
and consult Mel Wade before we make a treatment decision.
Without insecticides, we can potentially lose over 80% of our crops
to worms and stink bugs that may result in over $20,000 loss per
acre.” (Surveyed on May 30, 2012)

56. Albert Riddle, vegetable producer (Titus, AL): “We use the integrated
pest management recommendations provided by Chip East – our
Regional Extension Agent. I can lose over 50% of my tomato crop if I
did not follow IPM for insect and disease management. That is about
$10,000 loss per acre.” (Surveyed on May 23, 2012)

57. AND it all comes down to writing good
reports…
Questions for Dr. A?
Thank you for your patience!