Slides from 2007 on the design and evaluation of Anti-Phishing Phil, a game that teaches people how to avoid phishing attacks.
In this paper we describe the design and evaluation of Anti-Phishing Phil, an online game that teaches users good habits to help them avoid phishing attacks. We used learning science principles to design and iteratively refine the game. We evaluated the game through a user study: participants were tested on their ability to identify fraudulent web sites before and after spending 15 minutes engaged in one of three anti-phishing training activities (playing the game, reading an anti-phishing tutorial we created based on the game, or reading existing online training materials). We found that the participants who played the game were better able to identify fraudulent web sites compared to the participants in other conditions. We attribute these effects to both the content of the training messages presented in the game as well as the presentation of these materials in an interactive game format. Our results confirm that games can be an effective way of educating people about phishing and other security attacks.
Authors are Steve Sheng, Bryant Magnien, Ponnurangam Kumaraguru, Alessandro Acquisti, Lorrie Cranor, Jason Hong, and Elizabeth Nunge
Anti-Phishing Phil: The Design and Evaluation of a Game That Teaches People Not to Fall for Phish
1. CMU Usable Privacy and Security Laboratory
http://cups.cs.cmu.edu/
Anti-Phishing Phil
The Design and Evaluation of a
Game That Teaches People Not to
Fall for Phish
S. Sheng, B. Maginien, P. Kumaraguru,
A. Acquisti, L. Cranor, J. Hong, E. Nunge
2. • CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
Online game
• http://cups.cs.cmu.edu/antiphishing_phil/
Teaches people how to protect
themselves from phishing attacks
• Identify phishing URLs
• Use web browser cues
• Find legitimate sites with search engines
Anti-Phishing Phil
3. • CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
Why a game?
Security is a secondary task
Learning by doing
Fun and engaging
Better strategies
4.
5.
6.
7.
8.
9.
10. • CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
More about the game
Four rounds
• Increasing difficulty
• Two minutes in each round
Eight URL “worms” in each round
• Four phishing and four legitimate URLs
• Users must correctly identify 6 out of 8 URLs to
advance
11.
12. • CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
User Study
Test participants’ ability to identify phishing web
sites before and after training
• 10 URLs before training, 10 after, randomized
• Up to 15 minutes of training
Training conditions:
• Web-based phishing education
• Tutorial
• Game
14 participants in each condition
• Screened out security experts
• Younger, college students
13. • CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
14. • CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
15. • CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
Falling for Phishing
0.43
0.34
0.12
0.19 0.17
0.38
0
0.1
0.2
0.3
0.4
0.5
Existing training
materials
Tutorial Game
FalseNegativeRate
Pre test
Post test
16. • CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
Misidentifying Legitimate Sites
0.30
0.27
0.30
0.41
0.21
0.14
0
0.1
0.2
0.3
0.4
0.5
Existing training
material
Tutorial Game
FalsePositiveRate
Pre test
Post test
17. • CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
Results
Game group had the best performance
overall
Game group had fewest false positives
No significant difference in false negatives
among the three groups
18. • CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
Field Study
19.
20. • CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
Study Set-up
Test participants’ ability to identify phishing web
sites after training and the ability to retain the
knowledge
• 6 URL quiz
before training, after training, one week later
Conditions:
• Control
• Game
Completed training
• 423 in training group
• 292 in control group
21. • CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
Preliminary Results
31%
60%
92%
75%
81%
93%
0%
20%
40%
60%
80%
100%
Novice Intermediate Expert
Pretest
Post test
24. • CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
Deployment
We’ve released Phil under a Creative Commons
non-commercial license
Over the past few weeks we’ve been contacted
by several banks, retailers, other companies, and
government agencies who are interested in using
Phil in their employee training programs
• Can’t get employees to read security memos, but think
they will be willing to play a game and learn something
We’re working on setting up a commercial
licensing program, customized versions
25. • CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
Portuguese Version
26. • CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
Future Plans
Analyze field study results to understand
how game can be further improved
Continue to update game and use data
from public usage to evaluate and improve
Consider adding new modules to teach
different skills or reinforce skills through
alternate approaches
Consider special versions for kids, elderly,
specific brands, etc.
27. • CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
Acknowledgements
Members of Supporting Trust Decision
research group
Members of CUPS Lab
28. CMU Usable Privacy and Security
Laboratory
http://cups.cs.cmu.edu/
Play Anti-Phishing Phil:
http://cups.cs.cmu.edu/antiphishing_phil/
Notas do Editor
Good afternoon everyone, I am Steve Sheng from Carnegie Mellon University, I am part of the CUPS lab at CMU. Today, I will be talking about some of the work that we did at CUPS lab in order to find solutions to train users about phishing attacks. The work that I will be presenting today was jointly done with Bryant Maginien, Ponguru Kumaragu, Alessandro Acquisti, Lorrie Cranor, Jason Hong and Elizabeth Nunge.
Educating user have some constraints, The first constraint is that security is a secondary task, people are not visiting a website to look at its security features, they go to the website to complete transactions. Another constraint is people like learning by doing, they don’t like to sit down and read training materials. Education is more effective when users learn by doing rather than by learning the classroom instructions.
The scene: is sea, we have a small fish called Phil, her job is to eat all the worms.
So today, Phil swim by a worm, the worm is identified by a URL. A good worm is a legitimate URL, whereas a bad worm is a bait dropped by the phishers.
Phil needs to score 6 / 8 to move on to the next rounds, and the end of the round, phil got a chance to reflect what he missed.
In between rounds, we also have short tutorials to teach Phil better strategies to identify phishing. In this example, Phil’s father teaches Phil how to use a search engine.
STUDY WAS A THINK ALOUD STUDY that lasts 45 - 60 MINS.. WE CAREFULLY RECRUITED NON-EXPERTS USING THREE SPECIFIC QUESTIONS - THE DEIFNITION OF NON EXPERT IS THE SAME AS IN PREVIOUS STUDY THAT I SAID. It aimed at testing the participants’ ability to identify phishing websites. We presented them 10 websites before training, followed by a 15 minute break where users perform one of the three tasks: they read webased phishing education, they read the game tutorial, or they played the game. Users are randomly assigned in each of the conditions. There are fourteen non-expert participants in each condition, for a total of 42 participants.
All of them are statistical significant, there is no statistical difference between them in Either pre test or post test.
There are statistically different.
To summarize, there are -- No significant difference in false negatives among the three groups - Game group performed best in false positives - Game condition performed best in total correctness Effect between the tutorial and the game conditions not statistically significant. The next question we want to answer, is that is the increase in performance due to learning or raising awareness.