“Serious Games, Serious Play” Research Study: “Where Are the Opportunities?” By Michael Cai - Serious Games Conference 2012
•Transferir como PPT, PDF•
6 gostaram•3,128 visualizações
Michael Cai talks about “Serious Games, Serious Play” Research Study: “Where Are the Opportunities?” at Serious Games Conference 2012 ABSTRACT: This presentation aims to help define the nascent serious games industry and identify key opportunities and challenges. Research findings are based on in-depth interviews with more than 30 thought leaders, worldwide. Key discussion points include the following: • The overall landscape of the serious games industry • Growth dynamics in different industry segments • Market opportunities and challenges in various sectors • Cost for developing serious games • How the industry is currently addressing measurement and effectiveness
Recomendados
Mais conteúdo relacionado
Destaque
Destaque (20)
Semelhante a “Serious Games, Serious Play” Research Study: “Where Are the Opportunities?” By Michael Cai - Serious Games Conference 2012
Semelhante a “Serious Games, Serious Play” Research Study: “Where Are the Opportunities?” By Michael Cai - Serious Games Conference 2012 (20)
Mais de SeriousGamesAssoc
Mais de SeriousGamesAssoc (20)
Último
Último (20)
“Serious Games, Serious Play” Research Study: “Where Are the Opportunities?” By Michael Cai - Serious Games Conference 2012
- 1. Serious Games, Serious Play An Industry Overview Presented by: Michael Cai VP Research, Games and Technology at Interpret August 22, 2012
- 2. Date: 08/22/12 Study Specifications Presenter: M. Cai Key Discussion Points Overall landscape of the serious games industry Growth dynamics in different industry segments Market opportunities and challenges in various sectors Cost for developing serious games How the industry is currently addressing measurement and effectiveness Methodology Section: Survey Methods Telephone interview & 1 E-mail survey Page: Timing July/August 2012 2
- 3. Date: 08/22/12 Study Participants Presenter: M. Cai Interviews with Experts At least 10+ years in industry Hold positions such as: Section: 1 Page: 3
- 4. Date: 08/22/12 Sector Composition Presenter: M. Cai The experts’ experience covers all the major categories Section: 1 Page: 4
- 5. The Serious Games Landscape Defining the Market
- 6. Date: 08/22/12 Defining ‘Serious Games’ Presenter: M. Cai Common Themes: Section: 1 Page: 6 However…
- 7. Date: 08/22/12 Defining ‘Serious Games’ Presenter: M. Cai “There’s a tech answer and a marketing answer.” Section: 1 Page: 7
- 8. Opportunity Today vs. Three Date: 08/22/12 Presenter: Years Ago M. Cai Acceptance outpacing adoption Serious Games Acceptance Adoption Section: 1 “I don’t expect an upswing as the industry as a whole is Page: still pretty much in its growth stage, but there has been much greater improvement in the profile.” 8
- 9. Opportunity Today vs. Three Date: 08/22/12 Presenter: Years Ago M. Cai Reasons Acceptance is growing: • Stigma of • Traditional ways • More powerful ‘game’ is of training, technology vanishing education not • More tools • Gamers getting working available older • Games seen as Section: valid substitutes 1 Page: 9
- 10. Opportunity Today vs. Three Date: 08/22/12 Presenter: Years Ago M. Cai Reasons Adoption is falling behind: • The • Growth is • Employers economy niche- hesitant specific to spend Section: on the 1 workplace Page: 10
- 11. Date: 08/22/12 Size of Industry in Revenue Presenter: M. Cai “The answer to that question is very elusive, difficult to pin down. It’s hard to find research on the subject.” “There are indicators of growth, but it’s difficult to quantify.” “It’s all about definition. Should Section: Bejeweled count, since it’s known to 1 reduce stress? Or Wii bowling, for all the benefits in nursing homes.” Page: 11
- 12. Date: 08/22/12 Revenue Generation Presenter: M. Cai Direct Indirect Revenue generated Revenue not generated by the game itself by the game itself • Grants • Games for marketing • B2B contracts (used as assets to sell something else) • Sales training • Corporate (improvement in sales efficacy) Section: sponsorship • Process optimization 1 • Licensing software/ (process efficiencies gained) subscriptions Page: • Microtransactions (in virtual worlds) 12
- 13. Date: 08/22/12 Growth Sectors Presenter: M. Cai Agreement that there is: Section: 1 Growth in Growth in Healthcare Military Page: 13
- 14. Date: 08/22/12 Growth Sectors Presenter: M. Cai Differing opinions for Education and Corporate • Government/institutional • Game feature usage (leader involvement is growing boards, etc.) growing Section: • More money than ever • Companies have the budgets 1 Page: • Games lack production value • Economy causing frugality • Designers don’t know how to • Less likely to spend on 14 make educational games innovations such as games
- 15. Date: 08/22/12 Growth Trajectory Presenter: M. Cai “Much bigger” in the next 5 years due to: More gamers More platforms As gaming becomes the norm, barriers to entry get lower Section: 1 Page: 15
- 16. The Serious Games Landscape Costs and Timing
- 17. Date: 08/22/12 Average Cost of Game/Sim Presenter: M. Cai HUGE variation depending on sector and sales model Licenses; Education Per Classroom ~$100 Subscriptions; Healthcare Bundles $5K-$10K Range $30K-$500K; Corporate Average $65K Section: 2 Fixed Grants; Phase 1 $90K, Government Phase 2 $750K Page: $0 $500K $1M 17
- 18. Average Length of Date: 08/22/12 Presenter: Development Time M. Cai For all sectors, more complexity = more time Education 3 mos. Healthcare 6 mos. Corporate 3 mos. Section: 2 Government 2-3 yrs. Page: 18
- 19. Date: 08/22/12 Variables Impacting Cost Presenter: M. Cai Complexity Assets # of Platforms Music/Audio Desired Graphics Labor Section: 2 3D Testing Page: 19
- 20. Date: 08/22/12 Acceleration in Cost? Presenter: M. Cai Some say Yes, some say No Yes, making No, making games is games is more getting expensive! cheaper. • Level of complexity has • More off-the-shelf increased tools available Section: • Trying to make things • Licenses are cheaper 2 simple actually costs • Barriers of entry more are lower Page: • Cost of labor has grown 20
- 21. Date: 08/22/12 Types of Metrics Presenter: M. Cai Section: 2 Page: 21
- 22. Marketing, Game Development, Measurement Industry Challenges
- 23. Getting More Industries and Date: 08/22/12 Presenter: Schools to Use Games to Teach M. Cai Consider first: Section: 2 Page: 23
- 24. Getting More Industries and Date: 08/22/12 Presenter: Schools to Use Games to Teach M. Cai Position I: Providing Proof Publishing documentation that serious games do achieve desired training or educational outcomes Offering evidence of cost effectiveness, particularly in comparison to other methods of Section: training and learning 2 Page: 24
- 25. Getting More Industries and Date: 08/22/12 Presenter: Schools to Use Games to Teach M. Cai Position II: “Stealth Mode” Removing the stigma associated with the word “games” or taking it out of the name entirely Hiding “games” within the words “activity” Section: or “learning module” 2 Page: 25
- 26. Date: 08/22/12 Making Games More Effective Presenter: M. Cai Focus on the end user Understand Appeal to end user all types of Educate end requirements end users users • Avoid being • How do people • The game has “chocolate- want to play to make sense covered (PvP, PvE, to the user broccoli” etc.)? Section: 2 Page: 26
- 27. Games vs. Sims vs. Virtual Date: 08/22/12 Presenter: Worlds M. Cai Analyze the task/information that needs to be taught Don’t be afraid to combine! “Generally speaking, though, training games need to have a very strong simulation component while mere educational titles may indulge on a more Section: traditional gaming 2 experience.” Page: 27
- 28. Date: 08/22/12 Making Games More Effective Presenter: M. Cai Inherent to the game must be: Design Mechanics Games need to promote learning through their very mechanics, i.e., the act of playing = learning Point System Must assign points for participation, rewards; giving feedback, motivating, building a relationship Section: Stickiness 2 Pay attention to the psychological effects, find out what motivates people to play Page: 28
- 29. Date: 08/22/12 Making Games More Effective Presenter: M. Cai However, the following pose challenges to innovation: Dearth of Talented Designers “Good developers are usually lured to big studios or become indies… dedicated serious games companies might not find it easy to get the right talent.” Costs Complex design and realistic simulations are not cheap. Also, some institutions are dependent on Section: government funding. 2 Page: 29
- 30. Date: 08/22/12 Making Games More Effective Presenter: M. Cai And the biggest criticism: Need for Measurement Aren’t enough tools that are proven to actually measure the effectiveness of a game; progress is slow on that end. Section: 2 Page: 30
- 31. Improving Marketing to Date: 08/22/12 Presenter: Increase Buyer Interest M. Cai Emphasize Outcomes • Market the outcomes being achieved, make the focus of games secondary Positive Press • Publish more success Section: stories to improve 2 category’s image, show Page: positive side of games 31
- 32. Research Inspires… More Research Future Opportunities
- 33. Date: 08/22/12 The Ideal Trade Group Presenter: M. Cai Section: 2 Page: 33
- 34. Date: 08/22/12 Key Questions Raised by Our Presenter: Experts Wh e ga mob re M. Cai me il do s in f e the ? it are the What rs of ete es param ous gam a seri ? Is it et f mark gment o e? fra ng els thi some Is o ne ahea countr d of y game the ? Wh Section: oppo ere do Doe rt termi s 2 lie unities nol b the eyond matte ogy u r? susp sual Page: ects ? 34
- 35. THANK YOU!
Notas do Editor
- How would you define Serious Games? Common themes : Designed for a purpose beyond just entertainment Provides a useful outcome or essential service (not just recall back, but critical thinking and learning; raises awareness) Uses game technologies (hardware and software) Employs features commonly found in games – rewards, competition A number of interviewees cited differences of opinion on the definition among people within the same organization. “There’s a tech answer and a marketing answer.” Also disagreement about even the idea; “all games have the capacity for learning, a ‘serious’ game just wants to make that learning about a broader concept that can be exported from the game itself.” (paraphrased)
- How would you define Serious Games? Common themes : Designed for a purpose beyond just entertainment Provides a useful outcome or essential service (not just recall back, but critical thinking and learning; raises awareness) Uses game technologies (hardware and software) Employs features commonly found in games – rewards, competition A number of interviewees cited differences of opinion on the definition among people within the same organization. “There’s a tech answer and a marketing answer.” Also disagreement about even the idea; “all games have the capacity for learning, a ‘serious’ game just wants to make that learning about a broader concept that can be exported from the game itself.” (paraphrased)
- How do you see the market opportunity in the serious games industry today versus 3 years ago? “ The opportunity is growing, mainly because the stigma of the word ‘game’ is vanishing and gamers are getting older. Also, more serious games success stories mean more interest.” “ You can say the word ‘gaming’ out loud. We’ve gone through the ‘civil rights’ of gaming. Gamers have gotten older, are coming to work saying, ‘why can’t the interface look more interesting a fun like games.’ The demand is being put there by users.” “ In general when it comes to learning, whether K-12 or the workforce, people are realizing that the traditional ways of training and education are not working. Test results will say that people are prepared, but they don’t have thought skills or career skills. Games and situated learning allow them to create the connection between the content and the ‘so what.’” “ Potential clients have stopped comparing serious games to their ‘non-serious’ counterparts (entertainment games) but as valid substitutes for traditional training methods.” The industries that are using serious games (particularly healthcare) are more accepting today. There is still some resistance, but not much. Technology is now much more powerful; hyper-reality is more engaging. Tools are available (Facebook, etc.) that were not available before. There’s not just a push, but people are seeking games out now. “Doctors are saying, ‘can I play Bones in my few spare minutes on my cell phone.’” As the world becomes more digital, there is more opportunity to overlay games onto work. “ Serious games are being taken more seriously at a very rapid pace, but truly serious work is being curtailed by the short-sightedness and overwhelming emphasis on profit by corporations” (Same person, different question: “It’s not all about money. Ironically, if more enlightened motives for funding research existed, profits would be amplified prodigiously.”) Growth “hampered by the difficult, if not critical, macro-economic situation worldwide.” “ While the market is evolving, the adoption rate is still niche specific. I don’t expect an upswing as the industry as a whole is still pretty much in it growth stage but there has been much greater improvement in the profile.” A more pessimistic view: Workplace learning seems unchanged – people not willing to spend, relying on PowerPoint and social media Serious games as the future of learning? Results are incredibly inconclusive People are enchanted by games, but they are not always the answer just because they are a hot topic Example: You play the game, then stop the fun and go back to learning. Once initial enthusiasm is over, we need to figure out what makes a good game
- How do you see the market opportunity in the serious games industry today versus 3 years ago? “ The opportunity is growing, mainly because the stigma of the word ‘game’ is vanishing and gamers are getting older. Also, more serious games success stories mean more interest.” “ You can say the word ‘gaming’ out loud. We’ve gone through the ‘civil rights’ of gaming. Gamers have gotten older, are coming to work saying, ‘why can’t the interface look more interesting a fun like games.’ The demand is being put there by users.” “ In general when it comes to learning, whether K-12 or the workforce, people are realizing that the traditional ways of training and education are not working. Test results will say that people are prepared, but they don’t have thought skills or career skills. Games and situated learning allow them to create the connection between the content and the ‘so what.’” “ Potential clients have stopped comparing serious games to their ‘non-serious’ counterparts (entertainment games) but as valid substitutes for traditional training methods.” The industries that are using serious games (particularly healthcare) are more accepting today. There is still some resistance, but not much. Technology is now much more powerful; hyper-reality is more engaging. Tools are available (Facebook, etc.) that were not available before. There’s not just a push, but people are seeking games out now. “Doctors are saying, ‘can I play Bones in my few spare minutes on my cell phone.’” As the world becomes more digital, there is more opportunity to overlay games onto work. “ Serious games are being taken more seriously at a very rapid pace, but truly serious work is being curtailed by the short-sightedness and overwhelming emphasis on profit by corporations” (Same person, different question: “It’s not all about money. Ironically, if more enlightened motives for funding research existed, profits would be amplified prodigiously.”) Growth “hampered by the difficult, if not critical, macro-economic situation worldwide.” “ While the market is evolving, the adoption rate is still niche specific. I don’t expect an upswing as the industry as a whole is still pretty much in it growth stage but there has been much greater improvement in the profile.” A more pessimistic view: Workplace learning seems unchanged – people not willing to spend, relying on PowerPoint and social media Serious games as the future of learning? Results are incredibly inconclusive People are enchanted by games, but they are not always the answer just because they are a hot topic Example: You play the game, then stop the fun and go back to learning. Once initial enthusiasm is over, we need to figure out what makes a good game
- How do you see the market opportunity in the serious games industry today versus 3 years ago? “ The opportunity is growing, mainly because the stigma of the word ‘game’ is vanishing and gamers are getting older. Also, more serious games success stories mean more interest.” “ You can say the word ‘gaming’ out loud. We’ve gone through the ‘civil rights’ of gaming. Gamers have gotten older, are coming to work saying, ‘why can’t the interface look more interesting a fun like games.’ The demand is being put there by users.” “ In general when it comes to learning, whether K-12 or the workforce, people are realizing that the traditional ways of training and education are not working. Test results will say that people are prepared, but they don’t have thought skills or career skills. Games and situated learning allow them to create the connection between the content and the ‘so what.’” “ Potential clients have stopped comparing serious games to their ‘non-serious’ counterparts (entertainment games) but as valid substitutes for traditional training methods.” The industries that are using serious games (particularly healthcare) are more accepting today. There is still some resistance, but not much. Technology is now much more powerful; hyper-reality is more engaging. Tools are available (Facebook, etc.) that were not available before. There’s not just a push, but people are seeking games out now. “Doctors are saying, ‘can I play Bones in my few spare minutes on my cell phone.’” As the world becomes more digital, there is more opportunity to overlay games onto work. “ Serious games are being taken more seriously at a very rapid pace, but truly serious work is being curtailed by the short-sightedness and overwhelming emphasis on profit by corporations” (Same person, different question: “It’s not all about money. Ironically, if more enlightened motives for funding research existed, profits would be amplified prodigiously.”) Growth “hampered by the difficult, if not critical, macro-economic situation worldwide.” “ While the market is evolving, the adoption rate is still niche specific. I don’t expect an upswing as the industry as a whole is still pretty much in it growth stage but there has been much greater improvement in the profile.” A more pessimistic view: Workplace learning seems unchanged – people not willing to spend, relying on PowerPoint and social media Serious games as the future of learning? Results are incredibly inconclusive People are enchanted by games, but they are not always the answer just because they are a hot topic Example: You play the game, then stop the fun and go back to learning. Once initial enthusiasm is over, we need to figure out what makes a good game
- Consensus : “ The answer to that question is very elusive, difficult to pin down. It’s hard to find research on the subject.” “ There are indicators of growth, but it’s difficult to quantify. There are more people making games, but maybe not more money being made.” “ It’s all about definition. Should Bejeweled be counted as a serious game, since it’s known to reduce stress? Or how about Wii Bowling, for all the benefits in nursing homes.”
- Direct : Grants, for those who work with the government or in educational institutions Ad sponsorship Licensing software/subscriptions Transactions done within virtual worlds B2B contracts Corporate Support (especially educational institutions) Indirect (the game itself is not generating the revenue): Games for marketing – used as assets to sell something else Sales training – how well the training works Process optimization – how well you’re optimizing your process
- In what sector of the industry do you see the most growth? Education Games/Corporate sector, for training (for internal training?)/Health care games/Medical professional training/Government/Military sims/training/ Activist Games (games for good/games for change)/Any other sectors emerging? Strong consensus that there is growth in healthcare . “If you don’t call it a game, doctors are receptive to it.” “It’s a lot easier to talk with people about it now than two years ago. Business has picked up.” Mobile technology in particular is reshaping games in healthcare. Also big are health games and sports science (Kinect and Wii, Nike+, etc.) +2 Growth in military . +4 Decline in education . “Doesn’t usually have the production value. May not have the budget to pull off a 3D environment.” May be political. Disagreement: Big growth in education : most of the developers I talked to are education focused, and they say that government and institutional involvement is growing; more money than ever in the education sector. More educational games and platforms using gaming technology in schools in the UK. Increasing interest in the corporate sector , as game design features like achievements and leaderboards become widespread in training. +1 Also, corporations have the budgets to commission these types of projects. Disagreement: corporate sector is lagging behind , despite innovation programs, in tough economic times rather than revolutionize, corporations tend to hunker down. “Anything that is odd, strange, different, etc. will be looked at with disdain, despite the clear advantages.” “ Anything that takes a lot of time is the next candidate for growth.”
- In what sector of the industry do you see the most growth? Education Games/Corporate sector, for training (for internal training?)/Health care games/Medical professional training/Government/Military sims/training/ Activist Games (games for good/games for change)/Any other sectors emerging? Strong consensus that there is growth in healthcare . “If you don’t call it a game, doctors are receptive to it.” “It’s a lot easier to talk with people about it now than two years ago. Business has picked up.” Mobile technology in particular is reshaping games in healthcare. Also big are health games and sports science (Kinect and Wii, Nike+, etc.) +2 Growth in military . +4 Decline in education . “Doesn’t usually have the production value. May not have the budget to pull off a 3D environment.” May be political. Disagreement: Big growth in education : most of the developers I talked to are education focused, and they say that government and institutional involvement is growing; more money than ever in the education sector. More educational games and platforms using gaming technology in schools in the UK. Increasing interest in the corporate sector , as game design features like achievements and leaderboards become widespread in training. +1 Also, corporations have the budgets to commission these types of projects. Disagreement: corporate sector is lagging behind , despite innovation programs, in tough economic times rather than revolutionize, corporations tend to hunker down. “Anything that is odd, strange, different, etc. will be looked at with disdain, despite the clear advantages.” “ Anything that takes a lot of time is the next candidate for growth.”
- Growth Trajectory Not willing to project hard estimates (double, triple, etc.), but general agreement that it will be bigger based on: More people game (casual gamers, women, etc) More platforms (mobile, web) People who are familiar with games will become more prominent and numerous leading to: People in decision-making roles being comfortable with using games as tools Users being more willing and receptive to using games as training/learning tools Tools get simpler and cheaper; proliferation of engines and game-making suites make it easier and cheaper to hack together a game to see if it’s effective
- What is the average cost of a game/sim for the sectors you work in? (Add: and what length of development time is required?) Education Corporate training Health care Govt/Military Activist Games There is huge variation in cost and timing depending on sector and sales model (subscription, etc.). Corporate – cost depends on graphics and number of platforms desired. Anywhere from $30K-$500K, with an average around $65K. Timing about 3 months. For games done within company on own time, 1 full time person for a couple of weeks (~$10K). Healthcare – With the subscription model, sell bundles of $5-10K. But almost always have to sell custom contracts in addition to subscriptions. Timing, about 6 months. Government/military – payment in the form of grants, which are fixed. First phase $90K, second phase $750K, over the course of 2 years. The government is recognizing that things are more complex, are more willing to spend $1-$2M now. Timing is fixed too. Currently set at 2-3 years, but could be done faster, probably 9-18 months. Education – licensing: about $100 per classroom (35-40 kids), about $10-15 per household. 50,000 British pounds over 1 year in Japan. Small games take about a month or so with a small team; larger games/sims can take 3 months with a dedicated core team and auxiliary asset producers; some games (MMOs, etc.) are massive, multi-year projects costing lots of money. $75K - $500K. Etc. - “Basically, people do the best they can with the budget at their disposal and scope the project accordingly. Projects can take from a few weeks to a couple of years or even more.”
- What is the average cost of a game/sim for the sectors you work in? (Add: and what length of development time is required?) Education Corporate training Health care Govt/Military Activist Games There is huge variation in cost and timing depending on sector and sales model (subscription, etc.). Corporate – cost depends on graphics and number of platforms desired. Anywhere from $30K-$500K, with an average around $65K. Timing about 3 months. For games done within company on own time, 1 full time person for a couple of weeks (~$10K). Healthcare – With the subscription model, sell bundles of $5-10K. But almost always have to sell custom contracts in addition to subscriptions. Timing, about 6 months. Government/military – payment in the form of grants, which are fixed. First phase $90K, second phase $750K, over the course of 2 years. The government is recognizing that things are more complex, are more willing to spend $1-$2M now. Timing is fixed too. Currently set at 2-3 years, but could be done faster, probably 9-18 months. Education – licensing: about $100 per classroom (35-40 kids), about $10-15 per household. 50,000 British pounds over 1 year in Japan. Small games take about a month or so with a small team; larger games/sims can take 3 months with a dedicated core team and auxiliary asset producers; some games (MMOs, etc.) are massive, multi-year projects costing lots of money. $75K - $500K. Etc. - “Basically, people do the best they can with the budget at their disposal and scope the project accordingly. Projects can take from a few weeks to a couple of years or even more.”
- What are the variables (i.e., what impacts cost structure? Art assets, licensing, labor, etc.)? Graphics 3D Number of platforms desired Complexity – What’s being taught, and how intricate does the programming need to be Assets Music/Audio Testing Labor
- Has the cost of making serious games accelerated in recent years? Are there more off the shelf tools for building serious games? Some say yes, some say no : Yes because there are more tools available – more models, cheaper game engines, cheaper licenses; the barriers of entry are lower. No because the level of complexity has accelerated , trying to make things simple actually costs more. No because of the cost of labor . No because of off-the-shelf tools: “ There is a plethora of game engines, middleware and tools currently available today at little or no cost so it is actually possible to start building research prototypes and proofs of concept fairly cheaply. I'd say the cost of developing effective serious games that are not overly dependent on high end graphical assets or performance, has not increased significantly during the last few years.” “ Yes cost has definitely accelerated but in a good way, it has made the technology more accessible to a wider range of audiences.”
- “ Current state is paltry; I don’t think any measurement can be called state of the art presently. Many interesting articles but very few real studies and of the appropriate scale/time.” (The quote is from a professor working with serious games for education; in Education people have been more pessimistic) Build metrics into the game/sim : In many cases, the metric itself is part of the game/sim (i.e., have to meet the desired outcome in order to pass or move on). Observational/external metrics : An observer watches people playing the game and scores them. Or, the people who will be using the game test it and provide feedback. Sales data and other business revenue metrics : By how much did this increase revenue, what were the cost savings, what is the impact on the bottom line? Biometrics (among the university crowd): changes in heart rate, brain waves, facial expressions Data mining (i.e. games keep track of what you’re doing) “ Psychometrics is the field of study concerned with the theory and technique of psychologicalmeasurement, which includes the measurement of knowledge, abilities, attitudes, personality traits, and educational measurement. The field is primarily concerned with the construction and validation of measurement instruments such as questionnaires, tests, and personality assessments.” http://en.wikipedia.org/wiki/Psychometrics Kirkpatrick’s 4 Levels of Learning Evaluation http://en.wikipedia.org/wiki/Donald_Kirkpatrick#Four_Levels_of_Learning_Evaluation Four Levels of Learning Evaluation Kirkpatrick's four levels are designed as a sequence of ways to evaluate training programs. As you proceed through each of the levels, the evaluation becomes more difficult and requires more time and each level provides more information. Kirkpatrick notes in his book, 'Evaluating Training Programs (Third Edition)' that "none of the levels should be bypassed simply to get to level that the trainer considers the most important." [1] The four levels of Kirkpatrick's evaluation model essentially measure: Reaction - what participants thought and felt about the training (satisfaction; "smile sheets") Learning - the resulting increase in knowledge and/or skills, and change in attitudes. This evaluation occurs during the training in the form of either a knowledge demonstration or test. Behavior - transfer of knowledge, skills, and/or attitudes from classroom to the job (change in job behavior due to training program). This evaluation occurs 3-6 months post training while the trainee is performing the job. Evaluation usually occurs through observation. Results - the final results that occurred because of attendance and participation in a training program (can be monetary, performance-based, etc.) Several authors have suggested an addition of a fifth level of evaluation. JJ Phillips has argued for the addition of a "Return on Investment (ROI) level which is essentially about comparing the fourth level of the standard model to the overall costs of training. [2] Roger Kaufmanhas argued that ROI is essentially a level-four type of evaluation since it is still internal to the organization and that a fifth level of evaluation should focus on the impact of the organization on external clients and society. [3]
- What is required to get more industries/companies/schools to use games for training/learning? There are two main camps on this – education/validation, and re-naming/“stealth mode” Education/validation: Publish proof/evidence/case studies that serious games work. Convince the people making the decisions/holding the purse strings. A better appreciation of the potent psychological potentials inherent in serious games. “ A more conducive environment will naturally evolve when, in a few more years, more and more people grown up with games will start getting into high responsibility and management positions across all types of companies and agencies.” “ Culture – digital natives vs. immigrants. How comfortable are older professors/professionals with the technology and how they feel personally about utilizing it in their class or workplace.” Evidence of cost-effectiveness. Re-naming/“stealth mode”: Remove the stigma associated with the word games, or take the word “game” out of the name entirely. “Stealth mode” = games hidden inside the words “activity” or “learning module.” Change game publishers’ advertising practices on selling games. Even more than just the educational validation, the idea that measurement can be accurately implemented into games, then games can be designed based on measurement, resulting in a feedback loop that increases the effectiveness rapidly. Mindset: games are not always the answer, may be more hype in certain situations In education, for example: a game may be fun, but what did you learn by playing? Measurement needs to be improved. Educators always talk about measurement, but it’s all lip service Regarding the games themselves: Need more people who know what they’re doing, who understand games and how to synthesize content, who really understand behavioral motivation. Need good design. In some cases (Japan), easier access to creative tools is needed. “SL and OpenSim are either too controlled or too difficult. Unity 3D needs to be developed for a flatter learning path.” Another one (and this may also fit under challenges to making games/sims more effective) is time restraints.
- What is required to get more industries/companies/schools to use games for training/learning? There are two main camps on this – education/validation, and re-naming/“stealth mode” Education/validation: Publish proof/evidence/case studies that serious games work. Convince the people making the decisions/holding the purse strings. A better appreciation of the potent psychological potentials inherent in serious games. “ A more conducive environment will naturally evolve when, in a few more years, more and more people grown up with games will start getting into high responsibility and management positions across all types of companies and agencies.” “ Culture – digital natives vs. immigrants. How comfortable are older professors/professionals with the technology and how they feel personally about utilizing it in their class or workplace.” Evidence of cost-effectiveness. Re-naming/“stealth mode”: Remove the stigma associated with the word games, or take the word “game” out of the name entirely. “Stealth mode” = games hidden inside the words “activity” or “learning module.” Change game publishers’ advertising practices on selling games. Even more than just the educational validation, the idea that measurement can be accurately implemented into games, then games can be designed based on measurement, resulting in a feedback loop that increases the effectiveness rapidly. Mindset: games are not always the answer, may be more hype in certain situations In education, for example: a game may be fun, but what did you learn by playing? Measurement needs to be improved. Educators always talk about measurement, but it’s all lip service Regarding the games themselves: Need more people who know what they’re doing, who understand games and how to synthesize content, who really understand behavioral motivation. Need good design. In some cases (Japan), easier access to creative tools is needed. “SL and OpenSim are either too controlled or too difficult. Unity 3D needs to be developed for a flatter learning path.” Another one (and this may also fit under challenges to making games/sims more effective) is time restraints.
- What is required to get more industries/companies/schools to use games for training/learning? There are two main camps on this – education/validation, and re-naming/“stealth mode” Education/validation: Publish proof/evidence/case studies that serious games work. Convince the people making the decisions/holding the purse strings. A better appreciation of the potent psychological potentials inherent in serious games. “ A more conducive environment will naturally evolve when, in a few more years, more and more people grown up with games will start getting into high responsibility and management positions across all types of companies and agencies.” “ Culture – digital natives vs. immigrants. How comfortable are older professors/professionals with the technology and how they feel personally about utilizing it in their class or workplace.” Evidence of cost-effectiveness. Re-naming/“stealth mode”: Remove the stigma associated with the word games, or take the word “game” out of the name entirely. “Stealth mode” = games hidden inside the words “activity” or “learning module.” Change game publishers’ advertising practices on selling games. Even more than just the educational validation, the idea that measurement can be accurately implemented into games, then games can be designed based on measurement, resulting in a feedback loop that increases the effectiveness rapidly. Mindset: games are not always the answer, may be more hype in certain situations In education, for example: a game may be fun, but what did you learn by playing? Measurement needs to be improved. Educators always talk about measurement, but it’s all lip service Regarding the games themselves: Need more people who know what they’re doing, who understand games and how to synthesize content, who really understand behavioral motivation. Need good design. In some cases (Japan), easier access to creative tools is needed. “SL and OpenSim are either too controlled or too difficult. Unity 3D needs to be developed for a flatter learning path.” Another one (and this may also fit under challenges to making games/sims more effective) is time restraints.
- Several suggestions : Understanding end user requirements – what’s the problem that needs to be solved? A game may not always be the right solution, even if the client initially requests it. Some games are seen as “chocolate-covered broccoli” because they were created along the following lines: we’re trying to teach something, feel the need to make it fun, so we arbitrarily throw it into the game format. Need to think carefully about why what you’re trying to teach is fun, what is the right kind of experience of puzzle that matches learning points. Lots of derision from the people I interviewed for this type of design; games need to promote learning through the very mechanics, not as an afterthought. The act of playing it must teach a skill/knowledge that is translatable to the outside world, rather than just make it a vehicle for unassociated knowledge. “ If the context is not receptive to the game, even brilliant games will fail.” Educating end users – the game has to make sense to the user. Need to help people understand how they can use these tools. Cost – simulations, for example, are not cheap. K-12 institutions dependant on government funds price pressures against creativity, they must leverage existing resources and do not have enough to devote to truly effective games Appealing to all types of end users – this is particularly important internationally (i.e., in some countries the leaderboard is motivating and in others it is not). How do people want to play – player vs. self, environment, or player? Stickiness – paying attention to the psychological effects, finding out what makes people want to play for the sake of increasing their score, skill or knowledge. Successful games don’t always have to be 3D, FPS, etc. Some of the most popular games, such as Farmville, couldn’t get more simple. Figuring out what the point system is – points for participation, rewards, giving feedback, motivating, building a relationship. Talent with Experience – “ Good developers are usually lured into big studios or become indies to work on their own niche projects. Dedicated companies in the serious games field might not find easy to get the right talent. This is even more true for universities and research labs that may be forced to develop games with staff who had no real industry exposure and experience.” Teaching game design in schools. Measurement kept coming up; there aren’t enough tools that are proven to actually measure the effectiveness of a game, and progress is slow on that end. Some of the developers were confident in their games’ efficacy, but others (and particularly Eva Baker) were of the opinion that massive strides need to be made to really glean useful information about the effectiveness of serious games. Modularity: several developers mentioned the necessity to make games (but perhaps more accurately sims) as platforms, not just discrete products. Making an environment that either the developer can adjust or add to quickly, or maybe even the user can adjust to suit his needs, is a much more cost-effective solution overall.
- Obviously depends on the end result desired “ It’s often an AND (i.e., combination of game, sim, mobile, etc.), not an EITHER/OR.” In education, simulations may be better suited for service learning – through sim experiences participants see situations they’ve never encountered before, and are reacting to an environment modeled after a piece of the relevant world
- Several suggestions : Understanding end user requirements – what’s the problem that needs to be solved? A game may not always be the right solution, even if the client initially requests it. Some games are seen as “chocolate-covered broccoli” because they were created along the following lines: we’re trying to teach something, feel the need to make it fun, so we arbitrarily throw it into the game format. Need to think carefully about why what you’re trying to teach is fun, what is the right kind of experience of puzzle that matches learning points. Lots of derision from the people I interviewed for this type of design; games need to promote learning through the very mechanics, not as an afterthought. The act of playing it must teach a skill/knowledge that is translatable to the outside world, rather than just make it a vehicle for unassociated knowledge. “ If the context is not receptive to the game, even brilliant games will fail.” Educating end users – the game has to make sense to the user. Need to help people understand how they can use these tools. Cost – simulations, for example, are not cheap. K-12 institutions dependant on government funds price pressures against creativity, they must leverage existing resources and do not have enough to devote to truly effective games Appealing to all types of end users – this is particularly important internationally (i.e., in some countries the leaderboard is motivating and in others it is not). How do people want to play – player vs. self, environment, or player? Stickiness – paying attention to the psychological effects, finding out what makes people want to play for the sake of increasing their score, skill or knowledge. Successful games don’t always have to be 3D, FPS, etc. Some of the most popular games, such as Farmville, couldn’t get more simple. Figuring out what the point system is – points for participation, rewards, giving feedback, motivating, building a relationship. Talent with Experience – “ Good developers are usually lured into big studios or become indies to work on their own niche projects. Dedicated companies in the serious games field might not find easy to get the right talent. This is even more true for universities and research labs that may be forced to develop games with staff who had no real industry exposure and experience.” Teaching game design in schools. Measurement kept coming up; there aren’t enough tools that are proven to actually measure the effectiveness of a game, and progress is slow on that end. Some of the developers were confident in their games’ efficacy, but others (and particularly Eva Baker) were of the opinion that massive strides need to be made to really glean useful information about the effectiveness of serious games. Modularity: several developers mentioned the necessity to make games (but perhaps more accurately sims) as platforms, not just discrete products. Making an environment that either the developer can adjust or add to quickly, or maybe even the user can adjust to suit his needs, is a much more cost-effective solution overall.
- Several suggestions : Understanding end user requirements – what’s the problem that needs to be solved? A game may not always be the right solution, even if the client initially requests it. Some games are seen as “chocolate-covered broccoli” because they were created along the following lines: we’re trying to teach something, feel the need to make it fun, so we arbitrarily throw it into the game format. Need to think carefully about why what you’re trying to teach is fun, what is the right kind of experience of puzzle that matches learning points. Lots of derision from the people I interviewed for this type of design; games need to promote learning through the very mechanics, not as an afterthought. The act of playing it must teach a skill/knowledge that is translatable to the outside world, rather than just make it a vehicle for unassociated knowledge. “ If the context is not receptive to the game, even brilliant games will fail.” Educating end users – the game has to make sense to the user. Need to help people understand how they can use these tools. Cost – simulations, for example, are not cheap. K-12 institutions dependant on government funds price pressures against creativity, they must leverage existing resources and do not have enough to devote to truly effective games Appealing to all types of end users – this is particularly important internationally (i.e., in some countries the leaderboard is motivating and in others it is not). How do people want to play – player vs. self, environment, or player? Stickiness – paying attention to the psychological effects, finding out what makes people want to play for the sake of increasing their score, skill or knowledge. Successful games don’t always have to be 3D, FPS, etc. Some of the most popular games, such as Farmville, couldn’t get more simple. Figuring out what the point system is – points for participation, rewards, giving feedback, motivating, building a relationship. Talent with Experience – “ Good developers are usually lured into big studios or become indies to work on their own niche projects. Dedicated companies in the serious games field might not find easy to get the right talent. This is even more true for universities and research labs that may be forced to develop games with staff who had no real industry exposure and experience.” Teaching game design in schools. Measurement kept coming up; there aren’t enough tools that are proven to actually measure the effectiveness of a game, and progress is slow on that end. Some of the developers were confident in their games’ efficacy, but others (and particularly Eva Baker) were of the opinion that massive strides need to be made to really glean useful information about the effectiveness of serious games. Modularity: several developers mentioned the necessity to make games (but perhaps more accurately sims) as platforms, not just discrete products. Making an environment that either the developer can adjust or add to quickly, or maybe even the user can adjust to suit his needs, is a much more cost-effective solution overall.
- Several suggestions : Understanding end user requirements – what’s the problem that needs to be solved? A game may not always be the right solution, even if the client initially requests it. Some games are seen as “chocolate-covered broccoli” because they were created along the following lines: we’re trying to teach something, feel the need to make it fun, so we arbitrarily throw it into the game format. Need to think carefully about why what you’re trying to teach is fun, what is the right kind of experience of puzzle that matches learning points. Lots of derision from the people I interviewed for this type of design; games need to promote learning through the very mechanics, not as an afterthought. The act of playing it must teach a skill/knowledge that is translatable to the outside world, rather than just make it a vehicle for unassociated knowledge. “ If the context is not receptive to the game, even brilliant games will fail.” Educating end users – the game has to make sense to the user. Need to help people understand how they can use these tools. Cost – simulations, for example, are not cheap. K-12 institutions dependant on government funds price pressures against creativity, they must leverage existing resources and do not have enough to devote to truly effective games Appealing to all types of end users – this is particularly important internationally (i.e., in some countries the leaderboard is motivating and in others it is not). How do people want to play – player vs. self, environment, or player? Stickiness – paying attention to the psychological effects, finding out what makes people want to play for the sake of increasing their score, skill or knowledge. Successful games don’t always have to be 3D, FPS, etc. Some of the most popular games, such as Farmville, couldn’t get more simple. Figuring out what the point system is – points for participation, rewards, giving feedback, motivating, building a relationship. Talent with Experience – “ Good developers are usually lured into big studios or become indies to work on their own niche projects. Dedicated companies in the serious games field might not find easy to get the right talent. This is even more true for universities and research labs that may be forced to develop games with staff who had no real industry exposure and experience.” Teaching game design in schools. Measurement kept coming up; there aren’t enough tools that are proven to actually measure the effectiveness of a game, and progress is slow on that end. Some of the developers were confident in their games’ efficacy, but others (and particularly Eva Baker) were of the opinion that massive strides need to be made to really glean useful information about the effectiveness of serious games. Modularity: several developers mentioned the necessity to make games (but perhaps more accurately sims) as platforms, not just discrete products. Making an environment that either the developer can adjust or add to quickly, or maybe even the user can adjust to suit his needs, is a much more cost-effective solution overall.
- Emphasize outcomes – market the outcomes that are being achieved, make the glitz and glamour of games secondary. An evidence-based approach will win out. “ It’s more an issue of personal biases and unwillingness” Positive press – publish more success stories to improve the category’s image, show that there’s a positive side of games. Put the stories where the people are – social media, blogs, etc.
- Help us with the positive press. Promote our category. Come out with statistics, find stories and show that there’s successful adoption. Show us best of class. Find out where the breakthroughs are happening. Help us answer the questions we were asked that we don’t know the answers to – size of market space, size of category in terms of revenue, etc. Help us get into sectors that are tough to crack, or help us expand in sectors that are stuck in one type of tool only. Do research into peripheral devices – Kinect, Google glasses, etc. Develop a serious games locale within a virtual world. “Eat your own dog food.” Facilitate Research-Corporate Partnerships “Help in bringing together academic and industry players for possible partnership. Arrange events dedicated to particular fields within the serious gaming domain to attract clients who could potentially be interested in using serious games within their own organizations for specific purposes.” Set a standard: “I don’t think we need to build any more awareness as there are already in my opinion too many conferences on SG. What is needed is consolidation of the thoughts/findings of these conferences and coming up with a plan to examine assessment in the first instance and then standards in the second.” Research on measurement (Awareness already exists) Publication in good academic journals to spur research and promote high profile “ Be proactive in promoting serious games as means to serious goals, not as goals. ” In the long run, that implies getting rid of the term ‘serious games’.”
- What key questions did we miss? Does terminology matter? Does it matter if it’s called a game? What are the parameters of the serious games market? Does it exist? Is it a fragment of training or traditional games? Or interactive media? Who (besides people who work with the government) is making any money on this? Where are there opportunities beyond the usual suspects (government and healthcare). Or is it just the usual suspects? What tools are you using to build games? What platforms, etc.? Where do mobile games fit in? What is required to get people to use games for training and learning? What stops them from using it? Is America in the lead? If not, how can we become the lead? How are security and privacy being address? Within virtual worlds, how are you going to collect money and transactions? Within the virtual space? Or through a website? How do we establish productivity games as an additional sector within serious games? How can simulations from the private sector and the military inform education? Which journals or resources from the private sector and the military are most informative? (To be asked of consumers with children) Do your children or children in your family learn from games? Side effect learning from playing a particular set of games Specific school games Which do they prefer? Why do you think that is?