Craftworkz specializes in building applications that embed artificial intelligence. These applications can learn, discover, see, hear and interpret like humans. For several clients, Craftworkz has built custom chatbots using their own AI framework built in Python. To build this solution, Craftworkz has designed a microservices architecture that is deployed on IBM Bluemix. This highly available architecture consists of several Node.js and Python runtimes, Redis by Compose and several IBM containers. Depending on the application load, we can scale up or scale down the individual components. IBM Bluemix gave us the flexibility to use different compute options and easily bind off-the-shelf services with custom code.
Craftworkz at InterConnect 2017 - Creating a Highly Scalable Chatbot in a Microservices Architecture Using IBM Bluemix
1. Session ID: 3322A
Company: Craftworkz
Sam Hendrickx & Michiel Vandendriessche
IBM InterConnect 2017 Conference
Creating a Highly Scalable Chatbot in a Microservices Architecture Using IBM Bluemix
2.
3. Hi handsome! How are you doing
today?
Feeling great! Just visiting San
Francisco. How r u?
Not yet, we just came back from a
conference.
I’m doing fine. Any plans for tonight?
Want to go for a drink?
Yeah why not!
Awesome Michiel! Can you please
verify your identity here so I know
you’re not a creep: http://bit.ly/abc123
18. What can you tell me about IBM
InterConnect 2017?
InterConnect is a conference where
you can tap into the most advanced
cloud technology in the market
today!
Where is it being held?
The location of InterConnect is
Mandalay Bay, Las Vegas, NV
25. What can you tell me about IBM
InterConnect 2017?
InterConnect is a conference where
you can tap into the most advanced
cloud technology in the market
today!
Where is it being held?
I don’t know how to answer that
question…
65. SESSION
COMPOSE FOR MONGODB
API
MANAGER
API
MANAGER
API
MANAGER . . .
NODEJS RUNTIMES
COMPOSE FOR REDIS
PYTHON RUNTIMES
. . .
NLP
SESSION MANAGER
RESPONSE
CONNECTION MANAGER
NLP
SESSION MANAGER
RESPONSE
CONNECTION MANAGER
NLP
SESSION MANAGER
RESPONSE
CONNECTION MANAGER
66. SESSION
COMPOSE FOR MONGODB
API
MANAGER
API
MANAGER
API
MANAGER . . .
NODEJS RUNTIMES
COMPOSE FOR REDIS
PYTHON RUNTIMES
. . .SESSION MANAGER
RESPONSE
CONNECTION MANAGER
SESSION MANAGER
RESPONSE
CONNECTION MANAGER
SESSION MANAGER
RESPONSE
CONNECTION MANAGER
67. Get Connected
@craftworkz_co
Let’s stay in touch! We can meet you in one of our offices in
Leuven or Kontich. Or we can come to you. Nothing beats a coffee
and an interesting conversation ;-)
@craftworkz
@craftworkzco
@craftworkz_co
www.craftworkz.be
Veldkant 33A, 2550 Kontich
info@craftworkz.be
+32 498 44 50 61
Headquarter
Gaston Geenslaan 11 B4
BE-3001 Leuven
LEUVEN (BE) OFFICE
Veldkant 33A
BE-2550 Kontich
ANTWERP (BE) OFFICE
Pedro de Medinalaan 81,
NL-1086 XP Amsterdam
AMSTERDAM (NL) OFFICE
Notas do Editor
SAM
SAM
a couple of months ago -> San Francisco
When we came back from the conference -> Michiel Tinder
Michiel was swiping left & right
Came across beautiful girl
Match
Converstation went like this
SAM
SAM
SAM
- We we’rent the only ones
SAM
MICHIEL
Sam & Michiel from Belgium
Idea of where Belgium is -> animation
Now -> Las Vegas
MICHIEL
Very small country with only 11 million inhabitants
Very complex country
6 different governments in this very small country
3 different official languages
Over 1000 kinds of beer
** DOES ANYBODY HAVE AN IDEA WHICH LANGUAGES? **
MICHIEL
…
Era of chatbots
Build chatbots
A lot of cool tools for developers from different vendors on the market
But one problem…
MICHIEL
…
Those developer tools do not support the Dutch language
Very simple solution
MICHIEL
…
Let’s build this chatbot framework ourselves
MICHIEL
Founders of Craftworkz
Build innovative prototypes for other companies
Strong focus on artificial intelligence
Build an engine that can be trained to interpret sentences in a specific domain
Started with a very modest and humble approach
MICHIEL
Let’s take a look at this first approach
MICHIEL
REST API for handling HTTP requests
MICHIEL
NLP component for interpreting the sentences and invoking the right response
NLP in a chatbot context uses machine learning to learn how to interpret short sentences
Train the engine by giving examples of sentences and their corresponding interpretations
MICHIEL
Component that handles the actual responses
MICHIEL
Session component for keeping context
We deployed this onto Bluemix and we built a first chatbot that could have looked something like this
MICHIEL
We will do a live demo later in the session
SAM
SAM
SAM
SAM
SAM
MICHIEL
With this new architecture
A new problem arose
…
MICHIEL
Let’s take a look at the previous conversation
…
This only happened some times but not all the times
MICHIEL
** DOES ANYONE HAVE AN IDEA WHAT HAPPENED? **
MICHIEL
Take a look at the architecture
MICHIEL
Context is stored in memory in the instances
MICHIEL
Every engine has its own version of the context in memory and they are not in sync
What can you tell me about IBM InterConnect? -> handled by one engine
The follow up question: Where is it being held? -> handled by another engine not knowing about the context
MICHIEL
Solution: …
Looked at the Bluemix catalog
MICHIEL
Added a compose for MongoDB database
Shared across all the engines
Every engine now has a session manager for communicating with the MongoDB
SAM
SAM
SAM
SAM
SAM
SAM
SAM
SAM
SAM
SAM
MICHIEL
Chatbots need to integrate with external datasources
API’s, databases, SQL, NoSQL, CRM’s and so on
The responses can be enriched with data from these data sources
MICHIEL
…
MICHIEL
Added a connection manager to our engine
…
MICHIEL
Complete architecture looked something like this
Message comes from social media
Sent to the API
Pushed on the queue
Pulled from the queue
Analyzed
Response is generated, with external data
Keeping context in mind
Sent back to the frontend
MICHIEL
We encoutered a new problem
…
MICHIEL
Let’s take a look at how a message was processed before
Pulled from the queue to the engine, processed, a response in generated and sent back
MICHIEL
Now
Pulled from the queue
Processed
Response is enriched with external data -> communication takes some time
Sent back to the frontend
10 times as long
MICHIEL
10.000 users sending 10 messages in parallel
This was getting slow
Up to minutes response time
We didn’t really know how to solve this
MICHIEL
Installed an activity monitor plugin for bluemix
This is what it looked like under heavy load
Disk usage -> don’t really care
Memory -> seems okay
CPU is very idle
MICHIEL
Only 5% of CPU usage under heavy load
Most of the time, the CPU is waiting for I/O
Waiting for external data
MICHIEL
Multithreading to the rescue
Added 100 threads per instance working in parallel for handling messages
MICHIEL
If we now take a look at the activity monitor
MICHIEL
CPU usage of 93%
A lot more efficient
MICHIEL
Let’s take a look at the complete testing architecture
Jmeter for simulating 10.000users talking in parallel
Making requests to the API
MICHIEL
This is what it looked like in a testing environment