1. Ibrahim Fathy, Mostafa Aref, Omar Enayet, and Abdelrahman Al-Ogail
Faculty of Computer and Information Sciences
Ain-Shams University ; Cairo ; Egypt

2.  Introduction.
 Problem definition.
 Objectives.
 Intelligent OLCBP Agent Model.
 OLCBPRL Hybridization .
 Experiments and Results.
 Conclusion & Future Work.
Intelligent Online Case-Based Planning Agent Model for RTS Games – ISDA’10

3.  Online Case Based Planning is an architecture based on CBR.
 It addresses issues of plan acquisition, on-line plan execution,
interleaved planning and execution and on-line plan adaptation.
 Darmok is a previous system that applies this -without revision- to
RTS Games.
Intelligent Online Case-Based Planning Agent Model for RTS Games – ISDA’10

4.  A Reinforcement Learning Approach.
 Combines temporal difference learning technique “One-Step SARSA”
with eligibility traces to learn state-action pair values effectively.
 It’s an on-policy method.
Intelligent Online Case-Based Planning Agent Model for RTS Games – ISDA’10

5.  Considered a challenging domain due to: Severe Time Constraints –
Real-Time AI – Many Objects – Imperfect Information – Micro-Actions
Intelligent Online Case-Based Planning Agent Model for RTS Games – ISDA’10

6.  Research in learning and planning in real-time strategy (RTS)
games is very interesting.
 The research on online case-based planning in RTS Games does
not include the capability of online learning from experience.
 The knowledge certainty remains constant, which leads to
inefficient decisions.
Intelligent Online Case-Based Planning Agent Model for RTS Games – ISDA’10

7.  Proposing an intelligent agent model based on both online case-
based planning (OLCBP) and reinforcement learning (RL)
techniques.
 Increasing the certainty of the case base by learning from
experience.
 Increasing both efficiency and effectiveness of the plan
decision making process.
 Evaluating the model using empirical simulation on Wargus. ( A
clone of the well-known strategy game Warcraft 2)
Intelligent Online Case-Based Planning Agent Model for RTS Games – ISDA’10

8. Environment ( RTS Game “Wargus”)
Case Base
Offline
Phase (Case
Acquisition)
Plan
Expansion
/Execution
Module
Online Case-
Based
Learner
Behaviors
Goals Evaluated Case
Retrieved Case
Traces
Cases
Actions
Online
Phase
Intelligent Online Case-Based Planning Agent Model for RTS Games – ISDA’10

9. Intelligent Online Case-Based Planning Agent Model for RTS Games – ISDA’10

10. Case Goal Strategy Situation
State
Shallow Features Deep Features
Behavior
Pre-Conditions Alive-Conditions Success-Conditions
Snippet
Learning Parameters
Certainty Factor Eligibility Prior Confidence
Intelligent Online Case-Based Planning Agent Model for RTS Games – ISDA’10

11. Intelligent Online Case-Based Planning Agent Model for RTS Games – ISDA’10

12.  Since the evaluation of the case base is changed, the case retrieval
algorithm must change also.
 The case with the best predicted performance will be retrieved to be
executed.
Intelligent Online Case-Based Planning Agent Model for RTS Games – ISDA’10

13. Intelligent Online Case-Based Planning Agent Model for RTS Games – ISDA’10

15. Intelligent Online Case-Based Planning Agent Model for RTS Games – ISDA’10
Learning Parameter Value used
Learning Rate (α) 0.1
Discount Rate (γ ) 0.8
Decay Rate (λ ) 0.5
Exploration Rate 0.1

16. Intelligent Online Case-Based Planning Agent Model for RTS Games – ISDA’10

17. Intelligent Online Case-Based Planning Agent Model for RTS Games – ISDA’10

18.  Agent has learnt that building a smaller heavy army in
that specific situation (the existence of a towers
defense) is more preferable than building a larger
light army. Similarly, the agent can evaluate the entire
case base and learn the right choices.
Intelligent Online Case-Based Planning Agent Model for RTS Games – ISDA’10

19.  Online case-based planning was hybridized with
reinforcement learning in order to introduce an
intelligent agent capable of planning and learning
online using temporal difference with eligibility traces:
Sarsa (λ) algorithm.
 The empirical evaluation has shown that the proposed
model –unlike Darmok System - increases the
certainty of the case base by learning from experience,
and hence the process of decision making for selecting
more efficient, effective and successful plans.
Intelligent Online Case-Based Planning Agent Model for RTS Games – ISDA’10

20.  Implementing a prototype based on the proposed
model.
 Developing a strategy/case base visualization tool
capable of visualizing agent’s preferred playing
strategy according to its learning history. This will help
in tracking the learning curve of the agent.
 Finally, designing and developing a multi-agent system
where agents are able to share their experiences
together.
Intelligent Online Case-Based Planning Agent Model for RTS Games – ISDA’10

21. Thank You !
Questions ?
Intelligent Online Case-Based Planning Agent Model for RTS Games – ISDA’10