Teamtalk is a simple library to execute Pharo Smalltalk code in a cluster of Pharo Smalltalk instances. MapReduce is also supported. Based on Vert.x using the VerStix library. https://github.com/quentinplessis/Teamtalk

1. TEAMTALK
第109SMALLTALK勉強会
CLUSTER COMPUTING LIBRARY FOR PHARO SMALLTALK
Quentin Plessis
(カンタン プレシ)

2. TEAMTALK - CLUSTER COMPUTING LIBRARY FOR PHARO SMALLTALK
BACKGROUND
▸ Multi processing is not natively possible in Pharo Smalltalk
▸ To perform resource intensive operations, it is necessary to
split the load between several Pharo instances
▸ Efficiently setting up several Pharo instances is
troublesome
▸ Establishing communication between several Pharo
instances is tricky

3. TEAMTALK - CLUSTER COMPUTING LIBRARY FOR PHARO SMALLTALK
BACKGROUND
▸ Multi processing is not natively possible in Pharo Smalltalk
▸ To perform resource intensive operations, it is necessary to
split the load between several Pharo instances
▸ Efficiently setting up several Pharo instances is
troublesome
▸ Establishing communication between several Pharo
instances is tricky
Docker image
https://github.com/mumez/pharo-vnc-supervisor
VerStix (Vert.x based communication tool)
https://github.com/mumez/VerStix
https://www.slideshare.net/umejava/verstix

4. ▸ Pharo Smalltalk library making it possible to execute tasks
in a cluster of Pharos instances.
TEAMTALK - CLUSTER COMPUTING LIBRARY FOR PHARO SMALLTALK
TEAMTALK
VerStix
Teamtalk
Teamtalk Server (Vert.x)
Producer
Pharo A
VerStix
Teamtalk
Worker
Pharo B
VerStix
Teamtalk
Worker
Pharo C
VerStix
Teamtalk
Worker
Pharo D
https://github.com/quentinplessis/Teamtalk

5. ▸ Pharo Smalltalk library making it possible to execute tasks
in a cluster of Pharos instances.
TEAMTALK - CLUSTER COMPUTING LIBRARY FOR PHARO SMALLTALK
TEAMTALK
VerStix
Teamtalk
Teamtalk Server (Vert.x)
Producer
Pharo A
VerStix
Teamtalk
Worker
Pharo B
VerStix
Teamtalk
Worker
Pharo C
VerStix
Teamtalk
Worker
Pharo D
TASK 1 TASK 2 TASK 3
https://github.com/quentinplessis/Teamtalk

6. TEAMTALK - CLUSTER COMPUTING LIBRARY FOR PHARO SMALLTALK
PROTOCOL
AProducer
TASK 1
CB
Looking for Workers
Available!
Available!
Available!
Execute task
TASK 2Execute task
TASK 3Execute task
Executed TASK 1 RESULT 1
Executed TASK 2 RESULT 2
Executed TASK 3 RESULT 3

7. TEAMTALK - CLUSTER COMPUTING LIBRARY FOR PHARO SMALLTALK
TASK FLOW
AProducer
Execution code
Input
Result
Result process code
Task
[ :input | input + 1 ]
1
[ :result | result inspect ]
‘[ :input | input + 1 ]’
1
[ :input | input + 1 ]
1
22
[ :input | input + 1 ]
1
2
[ :result | result inspect ]
2

8. TEAMTALK - CLUSTER COMPUTING LIBRARY FOR PHARO SMALLTALK
REMARKS
▸ Support multiple producers
▸ Workers can be dynamically added / removed
▸ Pharo instances can be on different machines and use
different Pharo versions
▸ All communications are broadcasted throughout the
cluster
▸ Security issues are present but currently not dealt with

9. TEAMTALK - CLUSTER COMPUTING LIBRARY FOR PHARO SMALLTALK
GETTING STARTED
▸ Install Teamtalk in two Pharos images A and B
▸ Setup Teamtalk server (docker must be installed and running)
▸ [Pharo Image A] Create a producer
▸ [Pharo Image B] Create a worker
▸ [Pharo Image A] Add a task to execute
docker run -p 8080:8080 plequen/teamtalk-server
Metacello new
baseline: 'Teamtalk';
repository: 'github://quentinplessis/Teamtalk/pharo-repository';
load.
producer := TTProducer host: 'localhost' port: 8080.
worker := TTWorker host: 'localhost' port: 8080.
task := TTTask
executionCode: [ :input |
input + 1
]
resultProcessCode: [ :result |
result inspect.
]
input: 1.
producer addTask: task.

10. TEAMTALK - CLUSTER COMPUTING LIBRARY FOR PHARO SMALLTALK
TASK EXAMPLES
producer addTask: (TTTask executionCode: [ 1 inspect. 5 seconds wait. nil ]).
producer addTask: (TTTask executionCode: [ 2 inspect. 5 seconds wait. nil ]).
producer addTask: (TTTask executionCode: [ 3 inspect. 5 seconds wait. nil ]).
producer addTask: (TTTask executionCode: [ 4 inspect. 5 seconds wait. nil ]).
https://github.com/quentinplessis/Teamtalk
start := DateAndTime now.
number := 4.
i := 0.
number timesRepeat: [
task := TTTask
executionCode: [
10 seconds wait.
'OK' inspect.
'OK'
]
resultProcessCode: [ :result |
i := i + 1.
i = number ifTrue: [
(DateAndTime now asUnixTime - start asUnixTime) inspect
].
].
producer addTask: task.
].
task := TTTask
executionCode: [
ZnClient new get: 'https://www.google.com/'.
]
resultProcessCode: [ :result |
result inspect.
].
producer addTask: task.

11. TEAMTALK - CLUSTER COMPUTING LIBRARY FOR PHARO SMALLTALK
CLUSTER MANAGEMENT https://github.com/quentinplessis/Teamtalk#cluster-setup
▸ Install Teamtalk in a Pharo image Teamtalk.image
▸ Setup a Teamtalk server on port 8080
▸ Add consumers, potentially on different machines
▸ Add a producer
requires docker and ruby
export TEAMTALK_SERVER_PORT=8080
ruby spawner.rb --create-server $TEAMTALK_SERVER_PORT
ifconfig | grep inet
export TEAMTALK_SERVER_IP=.....
ruby spawner.rb --create-consumer --pharo-image ./Teamtalk.image --server-host
$TEAMTALK_SERVER_IP --server-port $TEAMTALK_SERVER_PORT
ruby spawner.rb --create-consumer --pharo-image ./Teamtalk.image --server-host
$TEAMTALK_SERVER_IP --server-port $TEAMTALK_SERVER_PORT
ruby spawner.rb --create-consumer --pharo-image ./Teamtalk.image --server-host
$TEAMTALK_SERVER_IP --server-port $TEAMTALK_SERVER_PORT
ruby spawner.rb --create-producer --pharo-image ./Teamtalk.image --server-host
$TEAMTALK_SERVER_IP --server-port $TEAMTALK_SERVER_PORT

12. TEAMTALK - CLUSTER COMPUTING LIBRARY FOR PHARO SMALLTALK
MAP REDUCE
TASK
SUB TASK
SUB TASK
SUB TASK
SUB TASK
SUB TASK
SUB RESULT
SUB RESULT
SUB RESULT
SUB RESULT
SUB RESULT
RESULT
MAP
REDUCESPLIT

13. TEAMTALK - CLUSTER COMPUTING LIBRARY FOR PHARO SMALLTALK
MAP REDUCE WITH TEAMTALK
▸ Examples: https://github.com/quentinplessis/Teamtalk#mapreduce
mapReduce := (TTMapReduce
splitBlockForInput: [ :input :tasksNumber |
"split input into sub inputs here"
]
mapBlockForSubInput: [ :subInput |
"process sub inputs here"
"map sub input to sub result"
]
reduceBlockWithCallback: [ :results :callback |
"reduce sub results here"
callback value: results
])
ttClientClass: TTProducer;
host: 'localhost';
port: 8080;
yourself.
mapReduce
input: { }
tasksNumber: 4
callbackDo: [ :result | result inspect ].

14. TEAMTALK - CLUSTER COMPUTING LIBRARY FOR PHARO SMALLTALK
IMPROVEMENTS
▸ Add labels to tasks for selective worker selection
▸ Propagate error handling from worker to producer
▸ Improve security
▸ Improve protocol to support additional languages
▸ Add a vote-based task scheduling system to remove the need for
producers to manage their tasks
▸ Add mechanism to handle loss of worker during task execution
▸ …

15. TEAMTALK - CLUSTER COMPUTING LIBRARY FOR PHARO SMALLTALK
TEAMTALK