2. What?
AUDIENCE RESEARCH
EVALUATION
VISITOR STUDIES
2
3. What? Observe
Data collection
Analysis
Polling
Research
Reporting
Surveys Feedback
Opinions
Understand
Improve
Who visits? 3
Marketing
4. What?
IN GENERAL…
Audience Research or Visitor Studies are
terms commonly used in the museum field to
describe the process of systematically
obtaining knowledge from and about
museum visitors, actual and potential, for the
purpose of increasing and utilizing that
knowledge in the planning and execution of
activities that relate to the public.
AAM Committee for Audience Research & Evaluation
4
5. What?
IN GENERAL…
Audience Research or Visitor Studies are
terms commonly used in the museum field to
describe the process of systematically
obtaining knowledge from and about
museum visitors, actual and potential, for the
purpose of increasing and utilizing that
knowledge in the planning and execution of
activities that relate to the public.
AAM Committee for Audience Research & Evaluation
5
6. What?
IN ITS SIMPLEST TERMS…
Evaluation is a process that
helps us answers 3 questions:
What?
SoWhat?
Now What?
6
11. So what?
OUTPUTS
• Projects
• Exhibits
• Collections
• Research
• Budgets
11
12. So what?
OUTCOMES. . . AKA IMPACTS . . .
I learned about . . .
My family is going to do . . .
I am more interested in . . .
I had fun. . . . We had fun. . . .
i.e. if successful, how will your audience be
12
different as a result of their experience?
13. So what?
BIG QUESTION:
HOW DO MUSEUMS MAKE A DIFFERENCE?
13
And how do we know?
14. So what?
NSF IMPACT CATEGORIES
Awareness, knowledge or understanding (of)
Engagement or interest (in)
Attitude (towards)
Behavior (related to)
Skills
GENERIC LEARNING OUTCOMES
Knowledge and Understanding
Enjoyment, Inspiration, Creativity
Attitudes and Values
Activity, Behavior and Progression
Skills
14
15. So what?
AWARENESS, KNOWLEDGE OR
UNDERSTANDING
Knowing what or about something
Learning facts or information
“Visitors will be aware that their
Making sense of something
daily actions have an impact on the
Deepening understanding
marine environment.”
How museums, libraries and archives operate
Making links and relationships between
15
16. So what?
ENGAGEMENT OR INTEREST
Participation, engagement, interest are
prerequisites for other types of learning
Having fun
“Children who participate in the
Being surprised
Innovative thoughts more engaged in
program will be
viewing contemporary art.
Creativity
Exploration, experimentation
Being inspired
16
17. So what?
ATTITUDES & VALUES
Feelings
Perceptions
Opinions about ourselves (e.g. self esteem)
“Adults will express a positive
Opinions or attitudes towards other people
Increased capacity for the importance
attitude towards tolerance of
Empathy play in their lives”
Increased motivation
Positive and negative attitudes in relation to an
experience 17
18. So what?
BEHAVIOR
What people do
What people intend to do
“Adults will avoid eating foods that
What people have done
have a substantially negative
Reported or observed actions
impact on the way people manage their
A change in
the environment.”
lives
18
19. So what?
SKILLS
Knowing how to do something
Being able to do new things
Intellectual skills
“Adults will be able to document
Information management skills
observations of plants that are
affected by climate change.”
Social skills
Communication skills
Physical skills 19
20. Now what?
WHAT DOES SUCCESS LOOK LIKE?
DEVELOPING INDICATORS
20
21. Now what?
FRYE ART MUSEUM
Impact Possible Indicators
1. Visitors will point, gesture, step
forward, nod, raise hand during
Visitors will be Engage physically tour program.
and verbally with the artwork
2. Visitors will ask questions,
make statements regarding the
artwork, and share opinions
with gallery guide or other
visitors
Evidence of success: % of visitors who ….
21
22. Now what?
YOUR TURN!
PROPOSED EXHIBIT TOPIC:
“HOMELESSNESS IN THE PACIFIC NORTHWEST”
1. Develop an impact statement, one for each category
2. Write your impact statements on the board
22
23. Now what?
YOUR TURN!
PROPOSED EXHIBIT TOPIC:
“HOMELESSNESS IN THE PACIFIC NORTHWEST”
1. As a class lets discuss some potential indicators of
success….
2. What evidence would show you that your desired impact
has been achieved.
23
24. Now what?
USE IT!
Australia Museum – “Misty” Cases
24
http://australianmuseum.net.au/Audience-Research-Blog/
25. Now what?
USE IT!
Seattle Aquarium – Harbor Seal Exhibit
25
26. SUMMARY
•Evaluation is a systematic process of collecting
information from your audience.
What?
•Evaluation should be included in all phases of
development.
•Museums should clarify their desired impacts
before they create their outputs.
So what?
•Evaluation can be use to help assess whether
desired impacts are achieved.
•Use evaluation to paint a picture of the museum
landscape, in decision making, and
Now what?
improvement of exhibits & programs.
26
•Don’t do it unless you’re ready to use it!
27. More!
RESOURCES
Visitor Studies Association (VSA)
www.visitorstudies.org
American Evaluation Association
www.eval.org
Committee for Audience Research & Evaluation (CARE)
www.care-aam.org
Inspiring Learning (GLO)
www.inspiringlearningforall.gov.uk
National Science Foundation
http://www.nsf.gov/pubs/2002/nsf02057/nsf02057.pdf 27
28. More!
JOURNALS & WEB
www.informalscience.org - evaluation reports
Visitor Studies
Curator: The Museum Journal
Journal of Interpretation Research
Journal of Museum Education
Museums & Social Issues
Informal Learning Review
28
Introduction, Me (MSU Museum, NYA, Museology & New Directions). My interest in VS, I want to know why things happen the way they do…and how we can improve our practice…Also, Seek to improve or practice through evidence. Evidence-based practice. SIMPLE CONCEPTS! Story about New York Aquarium, developing sea lion shows “you are an integral part of the environment around you and your actions can help or hurt that environment”… everyone always said “we have such an amazing conservation message”… my response: “how do we know that’s true?, how do we know that anyone is taking that away?” …. They’d say “well, if just one person gets it, we’ve done our job”…. After putting in so much time, money and effort into a program/exhibit… do we really want to just effect one person?.... I think we can shoot higher! And the way we do that is through gathering data, gathering evidence that what we’re doing is relevant, something visitors actually are responding to.
We’ll use these terms somewhat interchangeably.Ask group what comes to mind when they think of the term “Evaluation”.
These are some of the words you used when I asked about what you thought people who work with visitor studies do. Add in understand, and improve. Museum evaluators seek to understand and document the experiences of our visitors, and learning that occurs
The wordy definition …… AAM, committee for Audience Research & Evaluation
Important part here: systematic collection of information, visitors looking at actual and potential audiences, using that knowledge, and planning activities based on the data you collect.
Kathleen Tinworth, Audience Research Director and Denver Museum of Science & Nature. And easy philosophy to live by when thinking about starting to use evaluation in your work…. Easy for people to get without using so much jargon.
These are your results of your rated knowledge and experience…. Opinion Line: Not everyone has the same “knowledge” or “experience” with VS, but everyone comes with some perspective or opinion, it’s important to know how we feel, our own philosophy, before we undertake VS, or do it in our museums! Go through prompts.. Discuss responses – tell me more about why you think this, any brave souls want to play devil’s advocate? …. Evaluation is a valuable tool to determine an organization or institutions effectiveness.Evaluation should be everyone’s responsibility.Evaluation is best done by professionals with training and skills.As long as we provide a great visitor experience, we don’t really need evaluation.Politics determine decision making, not information we get from evaluation.Some data is better than no data.Evaluation take staff time and resources.It’s common to think evaluation is something we just tack on to the end of a project, or only do when we have extra time. But evaluation should be incorporated to the whole project development process…Also, Seek to improve or practice through evidence. Evidence-based practice. SIMPLE CONCEPTS! Story about New York Aquarium, developing sea lion shows “you are an integral part of the environment around you and your actions can help or hurt that environment”… everyone always said “we have such an amazing conservation message”… my response: “how do we know that’s true?, how do we know that anyone is taking that away?” …. They’d say “well, if just one person gets it, we’ve done our job”…. After putting in so much time, money and effort into a program/exhibit… do we really want to just effect one person?.... I think we can shoot higher! And the way we do that is through gathering data, gathering evidence that what we’re doing is relevant, something visitors actually are responding to.
When should you do evaluation? Stages of program or exhibit development. Evaluation isn’t just something you do at the end of a project… it’s an integral part of project development:Front End: conceptual, before the start, exploratory, visitor understanding, interest, misconceptions, paints a picture of the landscape before project begins, often called “needs assessment” (especially for grant proposals).Formative: Assess ongoing project activities as a program/exhibit is being developed (often iterative),Results are intended to offer direct, concrete, and practical ways to improve a project,Must be open to making changes midstream based on something that does not seem to be working.Remedial:Small studies that take place when program/exhibit is up and running,Tweaking and modifications, maybe after a “soft” open.Summative: To assess whether or not a project achieved its goals and objectives. Did the program/exhibit do what it was intended to do? Determine the efficacy of the program/exhibition,Serves to inform future exhibit and program developmentStory: back to aquarium, we wanted to do a show about voting…. But they said ,”maybe we shouldn’t because people already know that voting is a conservation action. Well, lets ask them! We did some front-end evaluation… they rated voting as lower importance so, we know that that’s an area we can beef up..… after the show, they rated it higher. A good example of evaluation in practice….
Breezing through this, some methods that can be used in evaluation. This is overly emphasized by museums looking to do evaluation. I.e. if we want to do evaluation, a common mistake is to jump right into methods without taking time to plan, “lets evaluate this exhibit, we should make a survey and hand it out to visitors”… . The method used should be dictated by the questions that we ask or the type of information we want to know.
SO WHAT!? Lets take a step back for a second, Brainstorm as group and write list on board of what museums do, is there a difference in the items on the board? Some about what we do (activities) some about the difference we make (outcome)
Outputs are the activities that museum undertake… they are the “products” or the “things” that museums do or create. It’s great to design and put up an exhibit, but “Why” would a museum put up an exhibit, develop a program?... That’s more important
Impacts (also referred to as outcomes) - The results of the outputs, the results of our activities… how are you effecting some sort of change in your audience? How do they think, feel, act differently… “at the end of the day I know this exhibit was successful when….”Go back to board. . . Which things were outputs, which were impacts? Describe this images (kids playing in a giant robin’s nest, what impact might the team have desired in creating this exhibit?
Impact categories & framework…. National Science Foundation CategoriesGeneric Learning Outcomes: From the Museums, Library and Archive Council in the UK: “Inspiring Learning” meant to help museums improve their learning activities
What are some of the outcomes we want for awareness, knowledge or understanding?
Similar to engagement, interest… but typically refers to more stable constructs like “empathy towards animals”… or happiness ratings scales, “agree-disagree” scale items… (like our opinion line)
Relevant to Zoo, Aquariums, Botanical Gardens – commonly, getting people to take part in conservation actions….For example, we might say we want to change behavior by saying youth who participate in the project will choose to spend more time outside in the next 3 months
Measurable demonstration of new skills or the reinforcement or practice of developing skills. (procedural aspects of knowing)
How do we know we’re making a difference, and how do we know we’re achieving these impacts? What does success look like? This brings us back to evaluation… evaluation comes in by identify Indicators of success…
With the person next to you, or at your table, develop impact statements for a proposed exhibit with the topic of homelessness in the pacific northwest
With the person next to you, or at your table, develop impact statements for a proposed exhibit with the topic of homelessness in the pacific northwest
Don’t conduct evaluation if you’re not prepared to use it! Some examples from practice… Australia Museum “Audience Research Blog” – Misty Cases example. Audience research there showed that people were sometimes disheartened by seeing multiple stuffed (taxidermied) animals in a row, such as these birds… exhibits was able to take that information and develop these “misty” cases where visitors can pass by without seeing them, or stand close to view just one or two… from a distance the cases appear frosted over.
New Directions team, doing a front-end evaluation of the current harbor seal exhibit…. Exhibit will change regardless, but an example of assessing visitor knowledge and attitudes toward harbor seals, and attention to the exhibit and it’s interpretive elements… Will be a great “baseline” or start, once renovation is complete, will re-do the evaluation – a summative – to see whether or not there is a change.
Summarize the discussion:
With the person next to you, or at your table, develop impact statements for a proposed exhibit with the topic of homelessness in the pacific northwest
With the person next to you, or at your table, develop impact statements for a proposed exhibit with the topic of homelessness in the pacific northwest
Contact information. Pass out sheets, quick evaluation, “what is one thing you took away, or one thing you think you’ll remember from our discussion today?”