This presentation shares how and why outcome mapping processes and principles enriched an internal self-evaluation process that was grounded in participatory action research methods toward deliberative democratic curriculum evaluation.
2. Schools as
Complex Adaptive
Social-Ecological
Systems The
Evaluation
Process
Teachers as
Curriculum
Evaluators
Outcome Mapping &
Internal Self-
Evaluation Planning
Potent
Possibilities
Presentation Roadmap
I.
II.
IV.
III.
V.
3.
4. Locating Evaluation Studies & Action Research
Usefulness
in facilitating
decision making &
transformations
Discovery
or validation of
generalizations
Hypothesis-
driven &
product-focused
Decision-
driven &
process-focused
Goals&Aims
Drivers & Focus
EVALUATION STUDIES
& ACTION RESEARCH
GENERAL
EDUCATIONAL
RESEARCH
9. Linear Perspective Nonlinear Perspective
Metaphor Machine Living System
Part & Whole
Relationship
Hierarchy
Sum of the Parts
Fractal Geometry
Emergence
Understanding
of Schools
Formal
Organizations
Learning
Organizations/
Communities
Cause & Effect
One-Way Causality
(Open-Loop)
Seeking Stability
Webs of Reciprocal
Causality (Closed-Loop)
Dynamic Equilibrium
Locus of Change External External & Internal
Schools as Social Systems
10. The Ecology of Educational Environments
Macro-
system
Exo-
system
Meso-
system
Micro-
system
Source:
Bronfenbrenner, 1976
11. Knowledge Cultures
as a Nested System Individual
Knowledge
Local
Community
Knowledge
Specialized
Knowledge
Organizational
Knowledge
Holistic
Knowledge
KEY
Collective
Thinking,
Learning,
& Inquiry
is
All the decision-making
knowledges
generating synergy
14. Schools as Complex Adaptive Systems
Generate
System-wide
Patterns
Agents
Interact Source: Patterson,
Holladay, & Eoyang, 2012
15. Schools as Complex Adaptive
Social-Ecological Systems (CASS)
Schools
as
CASS
Diverse &
Dynamic Agents
Non-Linear &
Unpredictable
Change & Co-
Evolution Rules
Flow of
Information
Feedback Loops
Nested
Systems
Structure
EmergenceInteraction
Dependent
but
Autonomous
Source:
Keshavarz,
Nutbeam,
Rowling, &
Khavarpour,
2010
16. Teachers as Curriculum Evaluators
Improvement
of Teaching
Improvement
of Learning
Political &
Social
Accountability
Professional &
Organizational
Development
Significant and Lasting Change is Teacher-Led
26. Influence Analysis
High
Low
Low High
LevelofPower
Level of Interest
Mixed Blessing Supportive
Marginal Non-Supportive
Monitor Defend Against
InvolveCollaborate With
KEY
Teacher
Admin
Support
Staff
Solid
Flexible
Not
Movable
T
A
SS
A
1
A
3
A
2
A
4
SS
1T
6
T
5
T
3
T
2
T
4
T
1
T
7
T
8
T
9
T
10
SS
2
T
11
28. Vision &
Mission
Teachers critically examine the civic curriculum and collaborate
to support the development of students’ civic competence.
Increased Civic Competency
Whole-
School
Effort
Support
Networks
&
Processes
Civic
Education
Resources
Informed
Teachers
Transdisciplinary
teacher teams promote
local and global civic competence using
authentic learning activities.
29. Mapping the Strategy
Individual/Collective
STRATEGIES
Environmental
STRATEGIES
Establish Newsletter and Good Citizen of the
Month Billboard (I-2)
Conduct Student and Teacher Surveys and
Interviews (I-1)
Organize School-wide Civic Activities (I-2)
Report
Civic
Findings
& News
to
Faculty
(E-2)
Implement
Observ-
ation
Teams
(E-3)
Establish
Critical
Friends
Groups &
Processes
(E-3)
Develop
Observ-
ation
Criteria
(E-1)
Facilitate Mock Election Workshops and Training (I-2)
Distribute
Civic
Lesson
Tips,
Plans, &
Materials
(E-2)
30. Cycle 2: Evaluation Planning
Determine
Evaluation
Questions
Establish
Timelines
Theory of Change
Chart the
Pathways
Plan
the Route
Identify
Information
Sources
31. Theory of Change Excerpt
The Project intends to see Social Studies
Teachers (SST) who recognize the importance
of increasing students’ civic competency
Teachers commit to action for increased student civic competency
Surveys Interviews
Report results
in workshops
80% SST participate
SST reflect
& engage in
generative
dialogue
SST
collab-
orate
on civic
curri-
culum
action
plans
Assumption
Goal
Outcome
Challenge
Strategy
Expect to See
Like to See
Love to See
KEY
“Hard” and “soft” data
inform about civic
competence levels,
curricular strengths,
and challenges
Increased
ownership &
responsibility
32. Cycle 3: Monitoring & Evaluation
Set
Monitoring
Priorities
Performance
Journal
Strategy
Journal
Analyze Cycle
0, 1, & 2 Data
Focus the
Monitoring
Document
the Changes
Outcome
Journals
33. Documenting Teacher Change
20% regularly facilitated
learning opportunities related
to civic competency
61% reported students
never researched local
community problems in class
44% reported they never
discussed civic responsibility
with students
72% regularly facilitated
learning opportunities related
to civic competency
30% reported students
never researched local
community problems in class
32% reported they never
discussed civic responsibility
with students
Prior to Project Two Years Into Project
34. Documenting Student Change
11% of students reported
that civic competency was
regularly addressed in Social
Studies class
62% reported they never
talked about the importance
of voting in Social Studies class
20% were able to correctly
identify the current mayor
Prior to Project Two Years Into Project
55% of students reported
that civic competency was
regularly addressed in Social
Studies class
16% reported they never
talked about the importance
of voting in Social Studies class
66% were able to correctly
identify the current mayor
35. Lessons Learned
More extensively targeting
all subject-matter teachers
More deeply engaging
Administrators
Building networks with
other area schools
Intensified whole-school
focus on civic competence
A network of educators
committed to civic education
that is less susceptible to
being dismantled
37. References
Aslin, H. J. & Brown, V. A. (2004). Towards whole of community engagement: A practical toolkit (MDBC Publication No. 14/04). Canberra City,
Australian Capital Territory: Murray-Darling Basin Commission.
Benson, T., LaVigne, A., Marlin, S., & Yates, J. (2010). Indicators of site readiness for innovation: Systems thinking in schools. Retrieved from
http://watersfoundation.org/wp-content/uploads/2013/08/Indicators-of-site-readiness.pdf
Brodnick, R. J., Jr. (2000). Conceptualizing social systems: A critical argument for the nonlinear perspective (Doctoral dissertation). Available from
ProQuest Dissertations and Theses database. (UMI No. 9965960)
Bronfenbrenner, U. (1976). The experimental ecology of education. Paper presented at American Educational Research Association Annual
Meeting, San Francisco, CA.
Brown, V. A. (2001). Planners and the planet. Australian Planner, 38(2), 67–73. doi:10.1080/07293682.2001.9657941
Brown, V. A. (2004). The more we are together... Collaborative decision making, social planning, and sustainability. Australian Planner, 41(3), 42–
45. doi:10.1080/07293682.2004.9982371
Brown, V. A., & Lambert, J. (2013). Collective learning for transformational change: A guide to collaborative action. New York, NY: Routledge.
Clarke, A. E. (2005). Situational analysis: Grounded theory after the postmodern turn. Thousand Oaks, CA: Sage Publications.
Cropper, B. (2003). Fact-file 2: Unpacking the 5 disciplines... Becoming a learning organisation. Retrieved from
http://www.thechangeforum.com/factfiles/FactFile_2-The_5_Disciplines.pdf
Crosby, D. A. (2014). In Imago Naturae: The ultimacy of nature as a closing of the gaps. In B. Donaldson (Ed.), Beyond the bifurcation of nature: A
common world for animals and the environment (pp. 180–191). Newcastle upon Tyne, UK: Cambridge Scholars Publishing.
de Guerre, D. W. (2001). Doing action research in one's own organization: An ongoing conversation over time. Systemic Practice and Action
Research, 15(4), 331–349. doi:10.1023/A:1016348421584
Dooley, K. J. (1996). A nominal definition of complex adaptive systems. The Chaos Network, 8(1), 2–3.
Dooley, K. J. (1997). A complex adaptive systems model of organizational change. Nonlinear Dynamics, Psychology, and Life Sciences, (1)1, 69–97.
doi:10.1023/A:1022375910940
Earl, S., Carden, F., & Smutylo, T. (2001). Outcome mapping: Building learning and reflection into development programs. Ottawa, Ontario,
Canada: International Development Research Centre.
Eguren, I. R. (2011). Theory of change: A thinking and action approach to navigate in the complexity of social change processes. The Hague, The
Netherlands: Humanist Institute for Development Cooperation (Hivos).
Eoyang, G. H. (2001). Conditions for self-organizing in human systems (Doctoral dissertation). Available from ProQuest Dissertations and Theses
database. (UMI No. 3040770)
38. References
Forrester, J. W. (1995). Counterintuitive behaviors of social systems. [based on October 7, 1970 testimony for the Subcommittee on Urban
Growth of the Committee on Banking and Currency, U.S. House of Representatives]. Retrieved from
http://static.clexchange.org/ftp/documents/roadmaps/RM1/D-4468-2.pdf (Original work published 1971)
Giancola, J. M., & Hutchison, J. K. (2005). Transforming the culture of school leadership: Humanizing our practice. Thousand Oaks, CA: Corwin
Press.
Greenwood, D. J., Whyte, W. F., Harkavy, I. (1993). Participatory action research as a process and a goal. Human Relations, 46(2), 175–192.
doi:10.1177/001872679304600203
Guba, E. G., & Lincoln, Y. S. (1989). Fourth generation evaluation. Thousand Oaks, CA: Sage Publications.
Hanberger, A. Democratic governance and evaluation. Paper presented at the Sixth European Evaluation Society Conference, Berlin, Germany.
Retrieved from http://www.edusci.umu.se/digitalAssets/66/66094_hanbergergovernance04.pdf
Henderson, J. G., & Gornick, R. (2007). Transformative curriculum leadership (3rd ed.). Upper Saddle River, NJ: Pearson Education.
Jones, H., & Hearn, S. (2009, October). Outcome mapping: A realistic alternative for planning monitoring, and evaluation. Overseas
Development Institute Background Note. Retrieved from http://www.odi.org/sites/odi.org.uk/files/odi-assets/publications-opinion-
files/5058.pdf
Kemmis, S., & McTaggart, R. (2005). Participatory action research: Communicative action and the public sphere. In N. K. Denzin & Y. S. Lincoln
(Eds.), The SAGE handbook of qualitative research (3rd ed., 559–604). Thousand Oaks, CA: Sage Publications.
Keshavarz, N., Nutbeam, D., Rowling, L., & Khavarpour, F. (2010). Schools as social complex adaptive systems: A new way to understand the
challenges of introducing the health promoting schools concept. Social Science and Medicine, 70(10), 1467–1474.
doi:10.1016/j.socscimed.2010.01.034
Krathwohl, D. R. (1998). Methods of educational and social science research: An integrated approach (2nd ed.). Long Grove, IL: Waveland Press.
Kurtz, C. F., & Snowden, D. J. (2003). The new dynamics of strategy: Sense-making in a complex and complicated world. IBM Systems Journal,
42(3), 462–483. doi:10.1147/sj.423.0462
MacBeath, J. (1999). Schools must speak for themselves: The case for school self-evaluation. New York, NY: Routledge.
MacDonald, B. (1976). Evaluation and the control of education. In D. Tawney (Ed.), Curriculum evaluation today: Trends and implications (pp.
125–136). London, UK: Macmillan.
Mendelow, A. L. (1981). Environmental scanning—The impact of the stakeholder concept. Proceedings of the International Conference on
Information Systems, USA, 2, 407–418. Retrieved from http://aisel.aisnet.org/icis1981/20
Merry, U. (1995). Coping with uncertainty: Insights from the new sciences of chaos, self-organization, and complexity. Westport, CT: Praeger.
39. Mertens, D.M. (2005). Research and evaluation in education and psychology: Integrating diversity with quantitative, qualitative, and mixed
methods (2nd ed.). Thousand Oaks, CA: Sage Publications.
Morgan, G. (2006). Images of organization (Updated ed.). Thousand Oaks, CA: Sage Publications.
Organizational Research Services. (2004). Theory of change: A practical tool for action, results and learning. Retrieved from
http://www.aecf.org/m/resourcedoc/aecf-theoryofchange-2004.pdf
Patterson, L., Holladay, R., & Eoyang, G. (2012). Radical rules for schools: Adaptive action for complex change. Circle Pines, MN: Human Systems
Dynamics Institute.
Patton, M. Q. (2002). Qualitative research and evaluation methods (3rd ed.). Thousand Oaks, CA: Sage Publications.
Richmond , B. (2004). An introduction to systems thinking. Lebanon, NH: isee systems.
Rittel, H. W. J., & Webber, M. M. (1973). Dilemmas in a general theory of planning. Policy Sciences, 4(2), 155–169. doi:10.1007/BF01405730
Roduner, D., & Hartmann, O. (2009, October). Module on step-0: How to prepare for OM Intentional Design. Outcome Mapping Ideas
(Paper No. 3). Retrieved from http://www.outcomemapping.ca/download/simonhearn_en_OMidea3.pdf
Savage, G. T., Nix, T. W., Whitehead, C. J., & Blair, J. D. (1991). Strategies for assessing and managing organizational stakeholders. Academy of
Management Executive, 5(2), 61–75. doi:10.5465/ame.1991.4274682
Schön, D. A. (1995). Knowing-in-action: The new scholarship requires a new epistemology. Change, 27(6), 26–34.
Senge, P. M. (2006). The fifth discipline: The art and practice of the learning organization (Revised ed.). New York, NY: Doubleday.
Sergiovanni, T. J. (1994). Building community in schools. San Francisco, CA: Jossey-Bass.
Shapiro, I. (2005). Theories of change. Beyond Intractibility (G. Burgess & H. Burgess, Eds.). Conflict Information Consortium, University of
Colorado, Boulder. Retrieved from http://www.beyondintractability.org/essay/theories-of-change
Themessl-Huber, M. T., & Grutsch, M. A. (2003). The shifting locus of control in participatory evaluations. Evaluation, 9(1), 92–111. doi:
10.1177/1356389003009001006
Wadsworth, Y. (1998). What is Participatory Action Research? Action Research International, Paper 2. Retrieved from:
http://www.scu.edu.au/schools/gcm/ar/ari/p-ywadsworth98.html
Watkins, A., & Wilber, K. (2015). Wicked and wise: How to solve the world’s toughest problems. Croydon, UK: Urbane Publications.
Zmuda, A., Kuklis, R., & Kline, E. (2004). Transforming schools: Creating a culture of continuous improvement. Alexandria, VA: Association for
Supervision and Curriculum Development.
References
40. Visual SourcesSlide 1: Cartoon evaluation image from Scheers & Wilson-Grau, 2008, GAN-Net Impact Community of Practice Meeting Presentation
Slide 3: Project Information graphic template from www.powerframeworks.com
Slide 4: Schools and Wicked Problems adapted from Rittel & Webber (1973); graphic template from www.SlideHunter.com
Slide 5: Locating Evaluation and Action Research adapted from Krathwohl (1998 )using positional mapping as described by Clarke (2005)
Slide 6: The Evaluation Process was adapted from Guba & Lincoln (1989) graphic template remixed from www.fppt.com
Slide 7: The Evaluation Cube was adapted from Schratz (1999) as cited in Themessl-Huber & Grutsch (2003) and MacBeath (1999)
Slide 9: Schools as Social Systems adapted from Brodnick, 2000, p. 120
Slide 11: Knowledge Cultures as a Nested System visual representation remixed from: solvingforpattern.org; Material from Valerie A. Brown
Slide 12: Cynefin Framework adapted from Kurtz & Snowden, 2004
Slide 13: Competent System Vs. Incompetent System from Zmuda, Kuklis, & Kline, 2004, p. 43
Slide 14: Complex Adaptive Systems quasicrystal graphic from http://wallpapers24k.blogspot.com/2012_05_01_archive.html networked agents
image from http://necsi.edu/research/networks/man53/man53.png
Slide 15: Schools as CASS graphic template remixed from www.fppt.com
Slide 16: Teachers as Curriculum Evaluators is adapted from from MacBeath (1999); graphic template from www.showeet.com
Slide 17: Inquiry Design Framework graphic template from www.SlideHunter.com
Slide 18: Participatory Action Research graphic template from www.showeet.com
Slide 20: Collective Learning Spiral graphic template remixed from www.slideteam.net; material from Aslin & Brown, 2004; Brown (2001, 2004);
Brown & Lambert (2013).
Slide 22: Democratic Evaluation Orientations graphic template remixed from www.showeet.com. Information adapted from Hanberger (2004)
Slide 23: Collective Inquiry Flow circular flow of process graphic template from www.slideteam.net
Slide 24: Cycle infinity ribbon graphic templates from www.powerframeworks.com
Slide 25: Schools and Systems Thinking graphic template from www.fppt.com
Slide 26: The Influence analysis representation fuses the insights of Mendelow (1981) and Savage, Nix, Whitehead, & Blair (1991), as a visual
representation of the positioning of the Project’s boundary partners.
Slide 28: Vision and Mission graphic template from www.fppt.com
Slide 29: Mapping the Strategy graphic template from www.SlideHunter.com
Slide 33: Documenting Teacher Change graphic template from www.slidehunter.com
Slide 35: Lessons Learned graphic template from www.powerframeworks.com
Slide 36: Teachers as curriculum evaluators handshake remixed from https://commons.wikimedia.org/wiki/File:Handshake2.svg ; Vertical Spiral
image from http://i2.wp.com/www.makeyourbestself.com/wp-content/uploads/2015/04/upwardspiral.jpg?resize=699%2C649
41. Outcome Mapping for Planning Evaluations in
American K–12 Urban Education: Potent
Possibilities by Tabia Lee is licensed under
a Creative Commons Attribution-NonCommercial
4.0 International License.
Please contact for permissions:
Tabia Lee, Ed.D.
916-588-7776, drtlee11@gmail.com
Notas do Editor
It is an honor to be here today to present to you why and how I have used outcome mapping to plan evaluations in urban education.
------------------**************-----------------------***************
At the time of the focal project of today’s presentation, I was a National Board Certified English and Social Studies Teacher and social studies department chairperson at an embattled urban school in East Los Angeles.
Participatory action research principles were complemented by Outcome Mapping principles and processes.
You can see here how the principles of outcome mapping are well suited for complex situations where there is a focus on development, learning, participation, and non-
linearity.
In this project, I made changes to the stages and processes of outcome mapping in order to fit the needs of the situation.
--------------------------------******************--------------------------*********************
Also, The design and monitoring stages of Outcome Mapping include
elements and tools that can be used in an ex-post evaluation to
study a strategy, an outcome, the results achieved by a particular
boundary partner, or an internal performance issue in greater
depth.