AIES 2021 Keynote lecture

Empathic Entertainment in Digital Game
Youichiro Miyake
AIES 2021
https://www.facebook.com/youichiro.miyake
http://www.slideshare.net/youichiromiyake
y.m.4160@gmail.com @miyayou

References
GAME AI PRO 3 (2017/6)
http://www.gameaipro.com/

Contents
• Chapter 1. Overview
• Chapter 2. Meta-AI
• Chapter 3. Meta-AI x ProceDural contents generation
• Chapter 4. Character AI
• Chapter 5. Character Behavior Learning
• Chapter 6. Decision Making Algorithm
• Chapter 7. Spatial AI
• Chapter 8. Deep Learning
• Chapter 9. Future of Game AI

Intelligent Game System
Game System
Intelligent
Game System
Transition in 1995 when 3D game begins.

What is digital game ?
• Interactive Digital Space
• The space becomes structured
• AI becomes a module in a whole system

Level Script
Navigation AI
Character AI
メタAI
1995 2000 2005 2010
(3D Games)
1994
2005
(
2010頃～
（Open World Game)
Spatial AI
1980
PlayStation
(1994)
Xbox360
(2005)
PlayStation3
(2006)
Scripted AI
三宅陽一郎、水野勇太、里井大輝、「メタAI」と「AI Director」の歴史的発展、日本デジタルゲーム学会（2020年、Vol.13, No.2）
LS-Model Model LCN-AI Model MCS-AI Cooperative
Model
MCN-AI Model

レベル
Character AI
Enemy
character
Player
情報獲得
Spatial AI
Meta-AI
Order
Ask &
Report
ゲーム全体をコントロール
Support
query
query
頭脳として機能
MCS-AI dynamic co-
operative model
Dynamic allocation of enemies
Observing level in real-time
Direction for agents
Making progress of game
Autonomous thinking
Cooperation
Team AI
Preparing data to make Meta-AI and
Character AI recognize the level
Managing object representation
Managing Navigation data
Path-finding / Tactical point analysis

AI in game, and AI outside of game
Game AI
（outside of game）
Game AI
（In game）
Meta-AI
Character
AI
Navigation
AI
Supporting
AI
QA-AI
Auto-balancing
AI
Interface-AI
Data Mining
Simulation
Game
Visualization
Biofeedback

Meta AI History
1980 1990
1980 1990 2000
Classical Meta AI
Modern Meta AI
Development of Character AI
(Autonomous AI )
Classical Meta AI to control difficulty of the game (Weaken enemies).
Modern Meta AI to design game system dynamically.

(Example） Xevious（Namco、1983）
Enemy Table Rewinding
Enemy 0
Enemy １
Enemy 2
Enemt 3
Enemy 4
Enemy 5

Classical Meta AI = Dynamic difficulty Adaptive AI
岩谷徹氏：ファンと一般のユーザーを満足させる方法のひとつは人工知能AIのような考え方です。プ
レーヤースキルをプログラム側から判断して、難易度を調整していくというものです。これを私はセ
ルフゲームコントロールシステムと呼んで10年以上前から開発に使っています。
－ International Game Designers Panel －
http://game.watch.impress.co.jp/docs/20050312/gdc_int.htm
Toru Iwatani (PACMAN Creator) said :
A way to satisfy game freak and game fun is AI .
By judging player’s skill by program, difficulty level is
dynamically adaptive to the player.
That is called “Level Control System”,
which had been used for 10 years.

AI Director (Meta AI) = Left 4 Dead
Michael Booth, "The AI Systems of Left 4 Dead," Artificial Intelligence and
Interactive Digital Entertainment Conference at Stanford.
http://www.valvesoftware.com/publications.html
Example : Left 4 Dead

Modern Meta AI
Meta AI dynamically changes a game
Meta AI
Allocation of
enemies
Spawning
enemies
Story
Generation
Procedural
Terrain Generation
User
Experience

Left 4 Dead (Valve)
AI Director controls NPC spawning distribution and game-play-pacing.
 Genre：Online Action
 Developer： Valve software
 Publisher : Valve software
 Hardware: XBox360
 Year： 2006 Using nav-mesh for recognizing and controlling
Real-time status of game.
Michael Booth, "Replayable Cooperative Game Design: Left 4 Dead," Game Developer's Conference, March 2009 http://www.valvesoftware.com/publications.html

Left 4 Dead – AI Director
Meta AI recognize game status in real-time
Predicting player’s route and spawning monsters dynamically.

AI Director decide when enemies are spawned.
USER’S INTENSITY
ACTUAL POPULATION
DESIRED
POPULATION
Build Up …プレイヤーの緊張度が目標値を超えるまで
敵を出現させ続ける。
Sustain Peak … 緊張度のピークを3-5秒維持するために、
敵の数を維持する。
Peak Fade … 敵の数を最小限へ減少して行く。
Relax … プレイヤーたちが安全な領域へ行くまで、30-45秒間、
敵の出現を最小限に維持する。
AI Director makes users relax, and break relax repeatedly.
Meta AI(=AI Director) observes players’ action in real-time in Left 4 Dead

Adaptive Dramatic Dynamic Pacing
[Basic Idea]
(1) Meta AI populates many enemies, enough to get
user’s intention up to the value (measured by
input).
(2) When user’s intention goes over the value, meta
AI stops the population.
(3) When user becomes relaxed, go to (1).
Michael Booth, "The AI Systems of Left 4 Dead," Artificial Intelligence and

Meta AI recognize game status
via Navigation-Mesh
Michael Booth, "The AI Systems of Left 4 Dead," Artificial Intelligence and Interactive Digital Entertainment Conference at Stanford.

Active Area Set
The area where meta-AI
populate and extinguish
agents is called
AAS（= Active Area Set）

Active Area Set

Flow Distance(predicting a path)
Meta- AI predict a path
- where they arrive
- where is back
- direction

Action to AAS
Meta AI spawns and extinguishes
enemies in Active Area Set.

Area seen from player
In the Area where player can see,
Meta-AI cannot spawn enemies

Area to spawn enemies
FRONT BACK
Meta- AI spawns enemies in the area players cannot see.

Monster and Item appearance frequency
High
Low
Wandereｒs (High)
Mobs(Middle)
Special Infected (Middle)
Bosses (Low)
Weapon Caches (Low)
Scavenge Items (Middle)

(1) Decide number of enemies to populate
by user’s escape route length.
(2) Populate enemies along the route in a
area around the player.
(3) When a player goes out the area, the
population stops, and enemies vanish.
(4) In the state of relax or when the are
can be seen from a player, all enemies
are eliminated from the map.
Algorithm of Meta AI

References
(1) Michael Booth, "Replayable Cooperative Game Design: Left 4 Dead," Game
Developer's Conference, March 2009.
(2) Michael Booth, "The AI Systems of Left 4 Dead," Artificial Intelligence and
(3) 三宅陽一郎, “メタAI”,「デジタルゲームの技術」
P.186-190, ソフトバンククリエイティブ

META-AI X PROCEDURAL
CONTENTS GENERATION
Chapter 3

Procedural Generation in WarFrame
The Living AI in Warframe's Procedural Space Ships Dan Brewer, Digital Extremes
http://aigamedev.com/open/coverage/vienna14-report/#session8

Black Combination in WarFrame

Procedural Generation Map in WarFrame

Auto-analyzing the map in WarFrame

Start, Exit, objectives auto-distribution
in WarFrame

Influence Map

Active Are Set

Meta AI
（AI Director, Dynamic Adaptive Pacing)

メタＡＩ（自動適応ペーシング）
Meta AI
（AI Director, Dynamic Adaptive Pacing)

Summary
Classical Game System Modern Game System
In classical game system, game contents are fixed after the development,
but in modern game system, game contents are dynamically created.

Summary
Classical Game System Modern Game System
Meta AI dynamically creates a game play flow, and gives an order to character AI and a game event.
The reason why meta AI action is simple is that character AI has become autonomous.

Referenced Papers
(1) Michael Booth, "Replayable Cooperative Game Design: Left 4 Dead," Game
Developer's Conference, March 2009.
(2) Michael Booth, "The AI Systems of Left 4 Dead," Artificial Intelligence and

FC SFC SS, PS PS2,GC,Xbox Xbox360, PS3, Wii
DC （次世代）
Hardware 時間軸
2005
1999
Character AI
Complex AI
AI and game world has become more complex
Simple AI
（スペースインベーダー、タイトー、1978年）（アサシンクリード、ゲームロフト、2007年）

(Ex) Space Invader (1978)
プレイヤーの動きに関係なく、決められた動きをする
（スペースインベーダー、タイトー、1978年）

（Ex） Prince of Persia
（プリンスオブペルシャ、1989年）

Halo (Bungie)
http://halo.bungie.net/inside/publications.aspx#pub15068

Environment
Intelligence
Artificial Intelligence＝ dynamically makes an AI’s action in harmony
with artificial environment.
What is intelligence？
Body
（Inner
Structure）
Input（Sensor） Output（Action）

Intelligence World
Environment World
Effector・Body
Sensor・
Body
Agent Architecture
Agent Architecture

Intelligence World
Environment World
Effector・Body
Sensor・
Body
Memory
Recog-
nition
Working
Memory
Small
Memory
Information Processing
Abstraction
of
Information
Decision
making
Body
Control
Effector・Body
Motion
Making
Motion synthesis process
body module
Motion
Synthesis
Decision making
Module
Decision making
Module
Decision making
Module
Agent Architecture

Intelligence World
Environment World
Effector・Body
Sensor・
Body
Memory
Recog-
nition
Working
Memory
Small
Memory
Abstraction
of
Information
Decision
making
Body
Control
Effector・Body
Motion
Making
body module
Motion
Synthesis
Decision making
Module
Decision making
Module
Decision making
Module
Objective・
Phenomenon
Agent Architecture

World
Sensor
知識
生成
Knowledg
e
Making
意思決定
Decision
Making
運動
生成
Motion
Making
Information Flow
記憶
Principle of Learning
Action
Result Decision
Making
Body

Deep Learning in Ubisoft
https://gdcvault.com/play/1025653/The-Alchemy-and-Science-of
The Alchemy and Science of Machine Learning for Games
Yves Jacquier （Ubisoft Montreal）

https://gdcvault.com/play/1025653/The-Alchemy-and-Science-of
35:00-
Deep Learning in Ubisoft
The Alchemy and Science of Machine Learning for Games
Yves Jacquier （Ubisoft Montreal）

Belief – Desire – Intention モデル
Desire
(Perceptrons)
Opinions
(Decision Trees)
Beliefs
(Attribute List)
Richard Evans, “Varieties of Learning”, 11.2, AI Programming Wisdom
Low Energy
Source =0.2
Weight =0.8
Value =
Source*Weight =
0.16
Tasty Food
Source =0.4
Weight =0.2
Value =
Source*Weight =
0.08
Unhappines
s
Source =0.7
Weight =0.2
Value =
Source*Weight =
0.14
∑
0.16+0.08+0.14
Threshold
（0～1の値に
変換）
hunger
Perceptron in Black & White （Lionhead,2000）

Black & White （Lionhead,2000）
Training of creature by learning
http://www.youtube.com/watch?v=2t9ULyYGN-s
http://www.lionhead.com/games/black-white/
14:20-

Reinforcement Learning
(Ex) Fighting Game
Kick
Punc
h
Magic
R_0 : Reward = Damage
http://piposozai.blog76.fc2.com/
http://dear-croa.d.dooo.jp/download/illust.html

LEARNING TO FIGHT T. Graepel, R. Herbrich, Julian Gold Published 2004 Computer Science
https://www.microsoft.com/en-us/research/wp-content/uploads/2004/01/graehergol04.pdf

3 ft
Q-Table THROW KICK STAND
1ft / GROUND
2ft / GROUND
3ft / GROUND
4ft / GROUND
5ft / GROUND
6ft / GROUND
1ft / KNOCKED
2ft / KNOCKED
3ft / KNOCKED
4ft / KNOCKED
5ft / KNOCKED
6ft / KNOCKED
actions
game
states
13.2 10.2 -1.3
3.2 6.0 4.0
+10.0
Ralf Herbrich, Thore Graepel, Joaquin Quiñonero Candela Applied Games Group,Microsoft Research Cambridge
"Forza, Halo, Xbox Live The Magic of Research in Microsoft Products"
http://research.microsoft.com/en-us/projects/drivatar/ukstudentday.pptx

Early in the learning process … … after 15 minutes of learning
Reward for decrease in Wulong Goth’s health

Early in the learning process … … after 15 minutes of learning
Punishment for decrease in either player’s health

Typical Agents in Digital Games
(1)Morikawa-kun (SCE、1997年）
(2)Astronoka (ENIX、1998年）
(3)The Sims（EA, Maxis, 2000）

1997「GANBARE MORIKAWAKUN 2GO」 (PLAYSTATION)
Character Control by
Neural Network
三宅陽一郎「ゲームAI技術20年の進化とこれから」 https://cedil.cesa.or.jp/cedil_sessions/view/1943 森川幸人「やってきたこと」より引用
Action
Player orders an action Player orders an action
Reaction from an item
Decision Making
Learning from the result
AI！

Color１
Color 2
Form
Smell
Sound
Behavior
…
Take an action
Knock
Kick
Take back
Eat
Jump
Push
おいしい
快い
不快
Impression from item
Nourishing
Enjoyable
Damaged
Fear
Fine
…
Table Data
…
…
…
…
Texture
Neural Network with
Backpropagation in game
ゲーム中にBP学習される
ニューラルネットワーク②
Put it on
the head

「Astronoka」（ENIX, 1998）
https://www.jp.square-enix.com/game/detail/astronoka/
Evolving by Genetic Algorithm

Genetic Algorithm
Make a group evolve in one direction
First generation New generation（１００～）
…
Evolution by generation
One generation produces a next generation

Genetic Algorithm
遺伝子
Gene
Next
Generation
Parent ①
Parent②
Pick up two parents
from base group
Crossover two genes Produce new generation
(selection) (crossover) (production)
Present Generation
The iteration makes a desired generation in the environment
遺伝子
Gene

Flow chart
of GA
A player makes traps
Beginning of a day
Trap battle begins
Trap battle ends
Evaluation
Fitness value
Order
Delete the last two NPC
Pick up two parents from NPCs with high fitness value
Generate two new NPCs
Generate new generation
Get enough number of new NPC
The day ends
Generation exchange
rate changes
Mutation rate changes

４－② Simulation and evaluation
Evaluation score is proportional to how long NPC go into the field
avoiding and breaking traps.
G
Fitness＝Score＋Time*0.3＋Enjoy param*0.5＋Trap_Score＋Safety＋HP*0.5
Time for clear
G
C B A B C
C B C
C
S
G
A
B
C
Start
Bonus =100
Goal
Bonus =50
Bonus =30
Bonus =10
Bonus =0

GA improvement in Game system
Average of fitness
A curve of fitness a day
Generations
(1) To give a feeling of evolution to users,
the game system evolves 5 generations a day by GA.
(2) To make constant speed of evolution of one day,
the game system controls the number of iterations of GA evolution.

The Sims（EA, Maxis, 2000）
• The game is to observe a society of agents
Ken Forbus, “Simulation and Modeling: Under the hood of The Sims” (NorthWerstern Univ)
http://www.cs.northwestern.edu/%7Eforbus/c95-gd/lectures/The_Sims_Under_the_Hood_files/frame.htm
Richard Evans, Modeling Individual Personalities in The Sims 3, GDC 2010
http://www.gdcvault.com/play/1012450/Modeling-Individual-Personalities-in-The

The Sims 「Motif Engine」
• AI Character Engine
Ken Forbus, “Simulation and Modeling: Under the hood of The Sims” (NorthWerstern University)
Data
- Needs
- Personality
- Skills
- Relationships
Sloppy - Neat
Shy - Outgoing
Serious - Playful
Lazy - Active
Mean - Nice
Physical
- Hunger
- Comfort
- Hygiene
- Bladder
Mental
- Energy
- Fun
- Social
- Room
Motive Engine
Cooking
Mechanical
Logic
Body
Etc.

Utility-based Agent
• Principle = Maximize a modd
Hunger +20
Comfort -12
Hygiene -30
Bladder -75
Energy +80
Fun +40
Social +10
Room - 60
Mood +18
- Urinate (+40 Bladder)
- Clean (+30 Room)
- Unclog (+40 Room)
Mood +26
- Take Bath (+40 Hygiene)
(+30 Comfort)
- Clean (+20 Room)
Mood +20
Bathtub
Toilet
Ken Forbus, “Simulation and Modeling: Under the hood of The Sims” (NorthWerstern大学、講義資料)

Mood calculation
Mood =
W_Hunger(X_Hunger) * X_Hunger + W_Engergy(X_Energy) * X_Energy + …
-100 0 100
-100 0 100 -100 0 100
-100 0 100
-100 0 100 -100 0 100
W_Hunger W_Energy
W_Comfort W_Fun
W_Hygiene W_Social
W_Bladder W_Room
-100 0 100 -100 0 100
Ken Forbus, “Simulation and Modeling: Under the hood of The Sims” (NorthWerstern大学、講義資料)

Utility
W_Hunger
X_Hunger
W_Hunger(-80)
-80 60
W_Hunger(60)

Utility comparison and The law of diminishing
marginal utility
Hunger degree at -80 ＝ W_Hunger(-80)*(-80)
Hunger degree at 60 ＝ W_Hunger(60)*(60)
Δ ＝ W_Hunger(60)*(60) - W_Hunger(-80)*(-80)
Utility for hunger
W_Hunger
X_Hunger
W_Hunger(-80)
-80 60
W_Hunger(60)
90
W_Hunger(90)

The law of diminishing marginal utility
Δ（-80 → 60）＝W_Hunger(60)*(60) - W_Hunger(-80)*(-80)
Δ (60→90）＝W_Hunger(90)*(90) - W_Hunger(60)*(60)
Δ（-80 → 60） is much larger than Δ(60→90）
Utility for hunger
W_Hunger
X_Hunger
W_Hunger(-80)
-80 60
W_Hunger(60)
90
W_Hunger(90)

CHARACTER
BEHAVIOR
LEARNING
Chapter 3-2

Intelligence World
Environment World
Effector・Body
Sensor・
Body
Memory
Recog-
nition
Working
Memory
Small
Memory
Abstraction
of
Information
Decision
making
Body
Control
Effector・Body
Motion
Making
body module
Motion
Synthesis
Decision making
Module
Decision making
Module
Decision making
Module

DECISION MAKING
ALGORITHM
Chapter 3-3

Decision Making Model
State-based AI
Goal-based AI
Rule-based AI
Behavior-based AI
Usually decision-Making is a very complex process.
But in artificial intelligence, there are some simple basic styles.
Simulation-based AI
Utility-based AI
Task-based AI
Decision Making
Reactive
Non-Reactive

F.E.A.R Agent Architecture
 Genre：Horror FPS
 Developer： Monolith Production
 Publisher : SIERRA
 Hardware: Windows
 Year： 2004
Agent Architecture Considerations for Real-Time Planning in Games (AIIDE 2005)
http://web.media.mit.edu/~jorkin/AIIDE05_Orkin_Planning.ppt
Sensors
Working
Memory
Planner
Blackboard
Navigation
Animation /
Movement
Targeting
Weapons
World
World

Planning by Chaining Example
kSymbol_
TargetIsDead
kSymbol_
TargetIsDead
Attack
kSymbol_
WeaponLoaded
kSymbol_
WeaponLoaded
Reload
kSymbol_
WeaponArmed
kSymbol_
WeaponArmed
Pick up
a weapon
None
Planner
Planning
Action
Pools

Replaning AI in F.E.A.R.
Jeff Orkins, Three States and a Plan: The AI of FEAR
http://alumni.media.mit.edu/~jorkin/gdc2006_orkin_jeff_fear.pdf

Decision Making Model
State-based AI
Goal-based AI
Rule-based AI
Behavior-based AI
Decision Making is generally a very complex high-degree process.
But for digital game there are 7 simple decision making algorithms.
Simulation-based AI
Utility-based AI
「(something)-based AI」means that an algorithm uses (something) as a unit.
Task-based AI
Decision Making

State Machine
State
State State
State Machine（Finite State Machine）
AI’s instruction is described in a state, and changes in the world and
AI are described in a transition condition.
A state machine has a loop structure but does not have feedback dynamics.

Hierarchical State Machine
Two enemy characters are patrolling. One player comes to the room.
AI character is described as Hierarchical Finite State Machine (HFSM).

(Example) Quake HFSM
http://ai-depot.com/FiniteStateMachines/FSM-Practical.html
Monster’s FSM
Spawn
Idle Attack
Die
Melee Melee
Smash Left Right
Finish spawning
Lost goal
Located goal
Zero health Zero health

Quake (id Software)
https://www.youtube.com/watch?v=LpEygM6ZkfI&t=350s

Watch
Chase Chase
Attack
Attack Warning
Command Ally
Watch the exit
Call an Ally
Ally not found
& Out of
battle field
Join with Ally
Join with Ally
Found Ally
Lost enemy
Found enemy
Lost enemy
Found enemy
Patrol
10 sec.
passed
Hear a
sound
Ally Responds
Transition
Condition State
Higher State
Hierarchical State Machine

Behavior-based AI
Behavior
A behavior represent not an animation detail
but a physical action.

Behavior-based AI
Behavior
Behavior
Decision-Making
Behavior
A behavior-based AI constructs its thinking by
using some behaviors.

(1) Hierarchical Structure of Box
(2) Each box has some behaviors
(3) Each box has one selection-rule
(random, sequential, priority-order…)
(4) Finally, one behavior is selected at end-of-tree.
(5) Decision Making
Behavior-based AI
 Genre：SciFi-FPS
 Developer： BUNGIE Studio
 Publisher : Microsoft
 Hardware: Xbox, Windows, Mac
 Year： 2004
Behavior-Tree (has become very popular method .)
Damian Isla (2005), Handling Complexity in the Halo 2 AI,
GDC Proceedings.,
http://www.gamasutra.com/gdc2005/features/20050311/isla_01.shtml

root
Battle
Retreat
Idle
Attack
Hide
Escape
Block
Stop
Recover
Trap
Sleep
Heal
Arrow
Sword
Hind in wood
Dig
Hide behind
object
Magic
Ice type
Wind type
Priority
Priority
Sequence
Sequence
Random
Priority
Random
Priority
Random
Behavior
（Leaf node）
Layer
Layer
Selection Rule
Selection Rule

State machine compared to Behavior tree
• State machine＝ Steady control step by step
• Behavior tree ＝ Adapt behavior fluently
We want to use both good points

Navigation Data
フリー素材屋Hoshino http://www.s-hoshino.com/

Navigation Data
Waypoint・Graph
（Networked Graph with points）
Navigation Mesh（Networked
Graph with polygons）
Walk-able
フリー素材屋Hoshino http://www.s-hoshino.com/

RTS demo
RTS - Pathfinding A*
https://www.youtube.com/watch?v=95aHGzzNCY8

TACTICAL POINT SYSTEM
Chapter 4-3

History of Tactical Point System
• Tactical Position Picking
Killzone (2005, Guerrilla) in Program
• TPS (Tactical Point System)
CRYENGINE (2010, CRYTEK) Tool & Runtime System
• EQS(Environment Query System)
UNREAL ENGINE 4 （2014, Epic games）Tool & Runtime System
• PQS (Point Query System)
FINAL FANTASY XV (2016, SQUARE ENIX)

Point Query System
A system to find a best positon
- for a character’s ability
- in a terrain
- in real-time

Point Query System principle
Point distribution （Generation）
distributing point around the objective （例）grid、circle
Filtering
Removing points not adjust for a purpose by a conditon (1)
．
．
．
Evaluation
Evaluation for remained points and pick up one points with best score
Filtering
Removing points not adjust for a purpose by a conditon (2)
Filtering
Removing points not adjust for a purpose by a conditon (N)

PQS (Point Query System)
A bowman finds the best point
(i) Game situation
(ii) Generating points around it
(iii) Filtering points with bad terrain
(iv) Filtering points where it’s arrow
can not reach
(v) Filtering points around buddies
(vi) Picking up one point with
highest terrain

敵
味方
今考えている
キャラクター
高台
高台
海
穴

Dota2 eSports
OpenAI Five: Dota Gameplay https://www.youtube.com/watch?v=UZHTNBMAfAA
解説：『Dota 2』における人間側のチャンピオンチームとAIチームの戦い
https://alienwarezone.jp/post/2316

[参考論文]
「OpenAI」については以下を参考にしました。
OpenAI Five
https://openai.com/projects/five/
Christopher Berner, et al.,“Dota 2 with Large Scale Deep Reinforcement Learning”
https://arxiv.org/abs/1912.06680

StarCraft~StarCraft2
（DeepMind, 2019）

StarCraftのAI
• Santiago Ontañon, Gabriel Synnaeve, Alberto Uriarte, Florian
Richoux, David Churchill, et al..
• “A Survey of Real-Time Strategy Game AI Research and
Competition in StarCraft”. IEEE Transactions on
Computational Intelligence and AI in games, IEEE
Computational Intelligence Society, 2013, 5(4), pp.1-19. hal-
00871001
• https://hal.archives-ouvertes.fr/hal-00871001

StarCraftの7つのアーキテクチャ (2010-2020)

StarCraft AI~StarCraft2 AI
（DeepMind, 2019）

[参考文献]
SC2LEについては以下の文献を参考にしました。
また図 7.8は以下の文献から引用しました。
Oriol Vinyals, et al., “StarCraft II: A New Challenge for Reinforcement Learning”,
https://arxiv.org/abs/1708.04782
PySC2 - StarCraft II Learning Environment
https://github.com/deepmind/pysc2

StarCraft II API
StarCraft II Binary
PySC2
Agent
アクション select_rect(p1, p2) or build_supply(p3) or …
観察
Resource
Possible action
Construction order
Screen
（Game
Information）
Minimap
Reward
-1/0/+1
SC2LE

評価値
Value
Network
Baseline features
アクション・タイプディレイユニット選択
命令発行ターゲット選択
Residual MLP MLP MLP Pointer
Network
Attention D
分散表現
MLP
分散表現
MLP
分散表現
MLP
Embedding
MLP
コア
Deep LSTM
スカラー
エンコーダー
MLP
エンティティ
エンコーダー
トランス
フォーマー
空間
エンコーダー
ResNet
ゲーム
パラメーター群
エンティティミニマップ

https://deepmind.com/blog/article/alphastar-mastering-real-time-strategy-game-starcraft-ii

DeepMind社「Capture the flag」
(2019)

シミュレーション
現実
機械学習
（ディープ
ラーニン
グ）
https://deepmind.com/blog/article/capture-the-flag-science

π
Game
Screen
Game
Screen
Game
Screen
Game
Screen
Game
Screen
Sampled
vector
Action
Inner Reward
ｗ
Winning or
Lose Judgemnet
Policy
Game
Point
Slow RNN
Fast RNN
Xt
𝑄𝑡 𝑄𝑡+1

赤フラグを青チーム陣地に
持ち帰る青エージェント

https://deepmind.com/blog/article/capture-the-flag-science

OpenAI「HIDE AND SEEK」
(2019年)

ボードゲーム自動生成
[参考文献]
Mark J. Nelson, “Bibliography: Encoding and generating videogame mechanics”,
IEEE CIG 2012 tutorial
https://www.kmjn.org/notes/generating_mechanics_bibliography.html
Cameron Browne,“Evolutionary Game Design”, SpringerBriefs in Computer Science, 2011
Cameron Browne, Frederic Maire, “Evolutionary Game Design”,
IEEE Transactions on Computational Intelligence and AI in Games ( Volume: 2, Issue: 1, March 2010)

game
Tic-Tac-Toe
Players
White
Black
board
end
All
Win
In-a-row
3
tiling
Square
i-nbors
shape
square
size
3 3
[参考文献]
IEEE Transactions on Computational Intelligence and AI in Games ( Volume: 2, Issue: 1, March
2010)

game
Y
Players
White
Black
board
end
All
Win
connect
all-sides
tiling
hex
shape
tri
regions
all-sides
size
11

game
Tic-Tac-Toe
Players
White
Black
end
All
Win
In-a-row
3
board
tiling
hex
regions
all-sides
size
11
shape
square

評価
選択
交叉
突然変異
ルール
チェック
整合性
がある
か？
ポリシー
選択
速度が
遅い？
ゴミ
箱
テストプ
レイ
前と似て
いる
引き分け
になり
やすい
母集団

[参考文献]
IEEE Transactions on Computational Intelligence and AI in Games ( Volume: 2, Issue: 1, March
2010)

Build = 1,11,46,Mine,8
Build = 1,11,48,Mine,16
Build = 1,11,47,Skyc,24
Build = 1,14,42,Turr,28
Build = 1,14,45,GrSl,100808
Build = 1,14,40,Turr,171184
Build = 1,9,47,Mine, 52
Build = 1,9,49,Mine, 60
Build = 1,12,44,RktL,100836
Build = 1,10,49,Drop,100812
Build = 1,13,45,RktL,100816
Build = 1,17,44,RktL,100820
Build = 1,10,45,Skyc,100824
Upgrade = 0,13,45,RktL,100820
Build = 0,11,46,Mine,8
Build = 0,11,48,Mine,16
Build = 0,11,47,Skyc,24
Build = 0,14,42,Turr,28
Build = 0,14,40,Turr,171184
Build = 0,9,47,Mine 52
Build = 0,9,47,Mine 60
Build = 0,15,45,GrSl,100808
Build = 0,10.,49,Drop,100812
Build = 0,13,45,RktL,100816
Build = 0,17,44,RktL,100820
Build = 0,10,45,Skyc,100824
Upgrade = 0,13,45,RktL,100816
Build = 0,17,49,Drop,100832

TT
L_1
TT
L2
TT
L3
TT
L4
TT
L1
TT
L2
TT
L3
TT
L4
親１親２
TT
L_1
TT
L2
TT
L3
TT
L4
TT
L_1
TT
L2
TT
L3
TT
L4
子１子２

City Control AI
Road obeserbing AI
Area Control AI
Observing and
Controlling AI
For building
Observing and
Controlling AI
For roads
Observing and
Controlling AI
For Park
Observing and
Controlling AI
For Human
Command Report Command Report
監視
制御
Huma
n Drone Robot
Digital
Avatar
Human
監視
制御
監視
制御
監視
制御
監視
制御
協調
協調
Coope
ration
Coopera
tion
監視
制御
監視
制御
監視
制御
City

City Control AI
Road obserbing AI
Area Control AI
Observing and
Controlling AI
For building
Observing and
Controlling AI
For roads
Observing and
Controlling AI
For Park
Observing and
Controlling AI
For Human
Command Report Command Report
監視
制御
Huma
n Drone Robot
Digital
Avatar
Human
監視
制御
監視
制御
監視
制御
監視
制御
協調
協調
Coope
ration
Coopera
tion
監視
制御
監視
制御
監視
制御
City
MetaAI
Character AI
Character AI
Character AI
Spatial AI

AIES 2021 Keynote lecture

Recomendados

Recomendados

Mais conteúdo relacionado

Mais procurados

Mais procurados (9)

Semelhante a AIES 2021 Keynote lecture

Semelhante a AIES 2021 Keynote lecture (20)

Mais de Youichiro Miyake

Mais de Youichiro Miyake (20)

Último

Último (20)

AIES 2021 Keynote lecture