I understand feeling tired. Make sure to take some time to rest and recharge. Is there anything else I can do for you?

1. Multiskill
Conversational AI
Mikhail Burtsev, PhD, Founder
Daniel Kornev, CPO
DeepPavlov.ai

2. DeepPavlov.ai
Why Multiskill is
important?
▪ Customer experience spans
multiple domains
• Surveys
• Promotions
• Campaigns
• Customer Service
• Technical Support
• …
▪ Every domain requires specific
skill
© Copyright PresentationGO.com
Pre-Purchase
Post-Purchase
Surveys
Promotions
Campaigns
Customer Service
Technical Support
Product Usage
Billing & Payment
Account Management
Logistics

3. DeepPavlov.ai
AliMe Assist
e-commerce assistant
▪ 3 Services with different ML
engines
• Assistance Service:
Slot Filling Engine
• Customer Service:
Knowledge Graph Engine
• Chatting Service:
Chat Engine
Li, F. L., Qiu, M., Chen, H., Wang, X., Gao, X., Huang, J., ... & Jin, G. (2017, November). Alime assist: An intelligent assistant for creating an innovative e-commerce
experience. In Proceedings of the 2017 ACM on Conference on Information and Knowledge Management (pp. 2495-2498). ACM.

4. DeepPavlov.ai
XiaoIce
socialbot
▪ 660 million users worldwide
▪ 5 countries
• China
• Japan
• USA
• India
• Indonesia
▪ 40 platforms
▪ Average CPS of 23 (better than
human to human)
Zhou, L., Gao, J., Li, D., & Shum, H. Y. (2020). The design and implementation of xiaoice, an empathetic social chatbot. Computational Linguistics, 46(1), 53-93.

5. DeepPavlov.ai
Conversational AI
Stack
Multiskill
orchestration
Conversational
skills
NLP frameworks
ML platforms
▪ Multiskill orchestration
• An AI assistant should use many skills to solve
different tasks
• Long term interaction with a user
▪ Conversational skills
• Conversational skill helps user with a specific
task
• Interaction with a user is limited by one dialog
▪ NLP frameworks
• Helps system to understand and generate
natural language
• Only a current user’s utterance or short
history of the dialog is processed
▪ ML platforms
• ML models training
Proprietary Open Source

6. DeepPavlov.ai
action =
request_location
system action
What we have right
now
▪ Modular dialog systems
• NLU (Natural Language
Undestanding)
• Detects Domain, Intent, Slots
(entities)
• DM (Dialogue Manager)
• Updates dialog state
• Decides the next action (policy)
• NLG (Natural Language
Generation)
• Generates response in natural
language
NLU
(Natural Language
Understanding)
• Domain detection
• Intent detection
• Entities detection
DM
(Dialogue Manager)
• Dialogue state
• Script, ML Policy
intent = request_movie
entities =
{ genre = ‘comedy’,
date = ‘weekend ’ }
semantic frame
NLG
(Natural Language
Generation)
• Generative models
• Templates
Are there any
comedy
movies to see
this weekend?
Where are
you?

7. DeepPavlov.ai
AI Assistant
Lifecycle
▪ Business as usual
• Product growth requires more and
more domains
• More domains – adding more features
and scriptspolicies
• Complexity ceiling
▪ DeepPavlov Vision
• Every domain is covered by separate
conversational skill (modularity)
• Every skill is deployed as a
microservice (scalability)
• Old skills can be reusedextended in
new product (interoperability)
• Separate skills might be developed
and maintained by separate teams
AGENT
MVP PRODUCT GROWTH MATURE AI ASSISTANT
domain skill
DeepPavlov
Vision
As
Usual
DM
NLU
feature
script
feature
script
feature
DM
NLU
feature
script
feature
script
feature
DM
NLU
feature
script
feature
script
feature
DM
NLU
domain skill
domain skill
default skill
default skill
default skill
AGENT
domain skill
domain skill
domain skill
default skill
default skill
default skill
domain skill
domain skill
feature
feature
script
feature
script
feature
script
feature
feature
feature script
feature script

8. DeepPavlov.ai
Agent
Models
/
Components
DeepPavlov solution
Skill Skill Skill
Chit-Chat
Factoid
Task-Oriented
√
√
Named Entity Recognition
√
√
Coreference resolution
√
√
Intent recognition
√
√
Insults detection
√
Q&A
√
√
Dialogue Policy
√
√
Dialog history
√
√
√
Lanaguage model
…
reddit
SQuAD
DSTC-2
Dataset
▪ NLP pipelines
• DeepPavlov Library provides
pretrained models and simple
declarative approach to build NLP
processing pipelines
▪ Conversational Skills
• DeepPavlov Dream is a collection of
pre-build conversational skills and a
default distribution package for
Dream AI Assistant
▪ Multiskill Orchestration
• DeepPavlov Agent is an engine for
conversational skill deployment and
orchestration

9. DeepPavlov.ai
DREAM socialbot
architecture
1. Multiple Annotators are used to
extract information from the
user input.
2. Skill Selector defines a subset of
active Skills based on the
extracted information.
3. Selected Skills propose their
response candidates.
4. Finally, Response Selector picks
a response to be sent to
Response Annotators and,
eventually, to the user.
All elements of the pipeline are
running asynchronously with two
points of synchronization: Skill
Selector and Response Selector.
Dialogue State serves as a shared
memory.

10. DeepPavlov.ai
DREAM socialbot
infrastructure
1. The core of the DREAM socialbot is implemented with
DeepPavlov Agent (DP-Agent) framework. It
orchestrates services for Skills, Annotators, Skill
Selector and Response Selector, and is located on AWS
EC2 instances with Docker Swarm.
2. Dialogue State history is stored on a separate instance
with MongoDB.
3. AWS Lambda performs HTTP requests to the DREAM-
agent by sending ASR tokens.
4. Testing infrastructure consists of Telegram bots for
interacting with the dev version of the socialbot or with
selected conversational skill only.
5. Dialogue analytics tool and dashboard were located on a
separate EC2 instance.
6. Cluster and application monitoring had configured alerts
to email and Slack.

11. DeepPavlov.ai
DeepPavlov
Ecosystem
▪ Open repository of NLP models
and pipelines
• easy to find and reuse NLP
components for development of new
skills or extension of existing
▪ Open repository of conversational
skills
• alternative implementations of the
most popular skills
▪ Open hub for AI Assistant
distributions
• general and domainindustry
specific distributions of skill sets

12. DeepPavlov.ai
DeepPavlov on NGC
ngc.nvidia.com/catalog/containers/partners:deeppavlov

13. DeepPavlov.ai
Conversational AI
DeepPavlov Stack
Multiskill
orchestration
Conversa-
tionalskills
NLP
frameworks
ML platforms
Proprietary Open Source
▪ Multiskill orchestration
• DeepPavlov Agent is an engine for
conversational skill deployment and
orchestration
▪ Conversational skills
• DeepPavlov Dream is a collection of pre-
build conversational skills and a default
distribution package for Dream AI
Assistant
▪ NLP frameworks
• DeepPavlov Library provides pretrained
models and simple declarative approach
to build NLP processing pipelines
▪ ML platforms
• TensorFlow and PyTorch as backends

14. DeepPavlov.ai
HOW TO BUILD MULTISKILL
AI ASSISTANT WITH DEEPPAVLOV
LIVING IN PANDEMICS
MAKES US LONELY
Wouldn’t it be nice to have a friend to care about us?

15. DeepPavlov.ai
MEET GERTY 3000
Can help Sam with problems on
the Moon Base Sarang?
Can entertain Sam? Yes ✔
Yes ✔
Can we emulate it with
DeepPavlov DREAM?
Yes ✔
Main Question
Functionality Analysis
Case: Gerty
In the Moon Movie

16. DeepPavlov.ai
Text OR
Voice
Input
TTS
(NeMo)
Deepy:
Example Architecture
Spell
Checking
NeMo ASR Harvesters
Status
Chit-Chat
(AIML)
Emotion
BUILT-IN
SKILL
SELECTOR
RULE-BASED
RESPONSE
SELECTOR

17. DeepPavlov.ai
services:
agent:
[..]
depends_on:
- mongo
harvesters_maintenance_skill:
[..]
mongo:
[..]
rule_based_response_selector:
[..]
nemo:
[..]
depends_on:
- agent
emotion_classification:
[..]
program_y:
[..]
spell_checking:
[..]
Deepy:
Docker Pipeline
Spell Checking
Annotators
Emotion Classification
Harvesters Status Skill
Skills
Chit-Chat Skill
NeMo ASR & TTS
Other Services
Rule-Based Response Selector

18. DeepPavlov.ai
services:
agent:
[..]
depends_on:
- mongo
harvesters_maintenance_skill:
[..]
mongo:
[..]
rule_based_response_selector:
[..]
nemo:
[..]
depends_on:
- agent
emotion_classification:
[..]
program_y:
[..]
clone_tts:
[..]
Deepy:
Part of Agent Pipeline
Annotators
Services Are in Groups
Skills
Response Annotators
Depend on groups (e.g., “skills”)
Services Are Isolated
Limited in what they see in dialog
Invoke Agent’s State Manager
Can run via HTTP or be Python-based
"skills": {
"harvesters_maintenance_skill":
{
"connector": {
"protocol": "http",
"url": "http://harvester
s_maintenance_skill:3002/respond"
},
"dialog_formatter": "dp_form
atters:full_dialog",
"response_formatter": "dp_fo
rmatters:base_skill_formatter",
"state_manager_method": "add
_hypothesis",
"previous_services": ["annot
ators"]
},
Response Selectors

19. DeepPavlov.ai
Deepy:
Harvesters M. Skill
def detect_intent(utterance):
[..]
return intent
def generate_response_from_db(intent, utterance):
[..]
return response, confidence
@app.route("/respond", methods=["POST"])
def respond():
[..]
dialogs = request.json["dialogs"]
for dialog in dialogs:
sentence = dialog['human_utterances'][-1]
['annotations'].get("spelling_preprocessing")
[..]
intent = detect_intent(sentence)
[..]
response, confidence = generate_response_from_db
(intent, sentence)
[..]
return jsonify(list(zip(responses, confidences)))
What is (all) harvesters’ status?
Intents
What is harvester status?
Prepare rover for a trip
Extracts post-processed sentences
Dialog Processing
Looks up for intents
If found, runs, & generates response
Returns response + confidence

20. DeepPavlov.ai
Deepy:
Chit-Chat Skill (AIML)
<?xml version="1.0" encoding="UTF-8"?>
<aiml version="2.0">
<category>
<pattern>I AM ^ TIRED</pattern>
<template>
🙁
<random>
<li>Get some sleep<get name="name"/>.
You're very tired.</li>
<li>Have a rest and be happy! How can
I help you?</li>
</random>
</template>
</category>
[..]
</aiml>
</xml>
Assistant Profile (Name, Place, etc.)
Patterns
Greeting scenario
Topics
Looks up for patterns
Dialog Processing
Picks random pre-defined response
If not sure, confidence is low (0.2)
Returns response + confidence