The main objective of this paper is to minimize the occluded areas in order to recognize the navigation of the surgeon’s tools for a two-arm autonomous robotic system for laparoscopic procedures. This robotic assistant needs the tracking of the surgeon’s surgical gestures in order to recognize the current maneuver and to execute the automated tasks of the robot. The surgical tools pose estimation is carried out by a Multiple Extended Kalman Filter (MEKF), where the movement models of the surgical tools depend on the maneuver which is being developed. This information is obtained by a maneuvers recognition system which is a part of the multimodal human machine interface (HMI) of the robot. The method proposed for reducing shadows has been applied to three in-vitro maneuvers which appear in the majority of the surgical protocols. The experiments show the behavior of this method for different time intervals of the occlusions.

1. SystemEngineeringand
AutomationDepartment
María Belén Estebanez Campos
belen@uma.es
System Engineering and Automation Department
http://www.isa.uma.es
University of Malaga (Spain)
INTELLIGENT ROBOTICS AND SYSTEMS (IROS 2011)
SURGICAL TOOLS POSE ESTIMATION FOR A
MULTIMODAL HMI OF A SURGICAL ROBOTIC
ASSISTANT
Belén Estebanez
Enrique Bauzano, Víctor Muñoz-Martínez

2. SystemEngineeringand
AutomationDepartment
María Belén Estebanez Campos
belen@uma.es
I. General Overview
ROBOTIC ASSISTANTS AND THEIR HUMAN MACHINE
INTERFACES
INDEX
I.Overview
II.Pose
Estimation
Model
III.Experiments
IV.Conclusions
DA VINCI (Niemeyer , 2000).
ERM Robot (Muñoz, 2006)
FAce MOUSe (Nishikawa, 2003).
LAPMAN de Medsys (Polet and Donnez, 2004).

3. SystemEngineeringand
AutomationDepartment
María Belén Estebanez Campos
belen@uma.es
I. General Overview
ROBOTIC ASSISTANTS AND THEIR HUMAN MACHINE
INTERFACES
INDEX
I.Overview
II.Pose
Estimation
Model
III.Experiments
IV.Conclusions
DA VINCI (Niemeyer , 2000).
ERM Robot (Muñoz, 2006)
FAce MOUSe (Nishikawa, 2003).
LAPMAN de Medsys (Polet and Donnez, 2004).
KaLAR system (Ko, 2007).

4. SystemEngineeringand
AutomationDepartment
María Belén Estebanez Campos
belen@uma.es
CISOBOT ROBOTIC ASSISTANT
INDEX
I. Overview
II.Pose
Estimation
Model
III.Experiments
IV.Conclusions
Microphone
Markers in surgical tools
E. Bauzano (IROS 2010)
I. General Overview

5. SystemEngineeringand
AutomationDepartment
María Belén Estebanez Campos
belen@uma.es
I. General Overview
PROBLEMS OF SURGICAL TOOLS TRACKING
INDEX
I.Overview
II.Pose
Estimation
Model
III.Experiments
IV.Conclusions
Disadvantages
• Existence of devices and surgical personnel in
the operation room
• Difficult location of the 3D sensors in the
operation room
• Specific tools movements during the surgical
task where the markers are hidden to the
sensors

6. SystemEngineeringand
AutomationDepartment
María Belén Estebanez Campos
belen@uma.es
HMI ARCHITECTURE
II. Pose Estimation Model
INDEX
I. Overview
II.Pose
Estimation
Model
III.Experiments
IV.Conclusions
ROBOTIC ASSISTANT
Movements Control
SURGEON
PATIENT
SURGEON
MODEL
POSE
ESTIMATION
MODEL
TRACKING 3D
VOICE
RECOGNITION
Position and
Orientation
Interpreted Voice
Maneuver
Voice
Command
Maneuver
Patient-Tool
InteractionPatient-Endoscope
Interaction
Surgical Tools
Movements
Position and
Orientation Estimated

7. SystemEngineeringand
AutomationDepartment
María Belén Estebanez Campos
belen@uma.es
SURGEON MODEL
II. Pose Estimation Model
INDEX
I. Overview
II.Pose
Estimation
Model
III.Experiments
IV.Conclusions
Insert Needle Hold Tissue Grab Needle
Maneuver: Specific sequence of basic actions, modeled by HMM.
Assist Knot Tie knot
Basic Action: Interaction between the surgical tools managed by
the surgeon.
Angle Distance Left Tool
Velocity
Right Tool
Velocity
Insert Needle
B. Estebanez (RAAD 2010)

8. SystemEngineeringand
AutomationDepartment
María Belén Estebanez Campos
belen@uma.es
SURGEON MODEL
II. Pose Estimation Model
INDEX
I. Overview
II.Pose
Estimation
Model
III.Experiments
IV.Conclusions
Maneuvers
Library
Intervention
Model
Data
Processing
RECOGNITION
SYSTEM
Position and
Orientation
of the
Surgical Tool
Maneuvers
HMMs
Last maneuver
Developed
Voice
Command
Current Maneuver
with Gestures
Command
CURRENT
MANEUVER
B. Estebanez (RAAD 2010)

9. SystemEngineeringand
AutomationDepartment
María Belén Estebanez Campos
belen@uma.es
POSE ESTIMATION MODEL
II. Pose Estimation Model
INDEX
I. Overview
II.Pose
Estimation
Model
III.Experiments
IV.Conclusions
Insert Needle Hold Tissue Grab Needle Extract Tool
e2 eNe1
Maneuver is divided into characteristic movements. A Characteristic Movement
(ei) is a well defined gesture inside the maneuver.
CHARACTERISTIC
MOVEMENTS
Selection (ei)
ESTIMATION
FUNCTIONS
Selection (fj)
Position and
Orientation
of the
Surgical Tool
Current
maneuver
Pose Estimation Model is a Multiple Extended Kalman Filter where their
Estimation Functions are selected according to the Current Maneuver and its
last Characteristic Movement.
Position
and
Orientation
Estimated
e3 ei

10. SystemEngineeringand
AutomationDepartment
María Belén Estebanez Campos
belen@uma.es
III. Implantation and Experiments
IMPLANTATION: CUTTING MANEUVER
2 4 6 8 10 12
80
100
120
Y(mm)
2 4 6 8 10 12
-1350
-1300
-1250
time (sec)
Z(mm)
2 4 6 8 10 12
0
20
40
60
Y(mm)
2 4 6 8 10 12
-1400
-1350
-1300
-1250
time (sec)
Z(mm)
2 4 6 8 10 12
80
100
120
Y(mm)
2 4 6 8 10 12
-1350
-1300
-1250
time (sec)
Z(mm)
2 4 6 8 10 12
0
20
40
60
Y(mm)
2 4 6 8 10 12
-1400
-1350
-1300
-1250
time (sec)
Z(mm)
INDEX
I. Overview
II.Pose
Estimation
Model
III.Experiments
IV.Conclusions
e1 e2 e3 e4 e3 e3 e5 e1 e2 e6 e1 e6 e1 e5
Surgeon’s Left Hand Surgeon’s Right Hand

11. SystemEngineeringand
AutomationDepartment
María Belén Estebanez Campos
belen@uma.es
III. Implantation and Experiments
EXPERIMENTS: CUTTING MANEUVER
INDEX
I. Overview
II.Pose
Estimation
Model
III.Experiments
IV.Conclusions
2 4 6 8 10 12 14
250
300
350
400
450
Surgeon's Left Hand
X(mm)
2 4 6 8 10 12 14
60
80
100
120
140
Y(mm)
2 4 6 8 10 12 14
-1400
-1350
-1300
-1250
time (sec)
Z(mm)
2 4 6 8 10 12 14
250
300
350
400
450
Surgeon's Right Hand
X(mm)
2 4 6 8 10 12 14
-20
0
20
40
60
80
Y(mm)
2 4 6 8 10 12 14
-1400
-1350
-1300
-1250
time (sec)
Z(mm)
2 4 6 8 10 12 14
250
300
350
400
450
Surgeon's Left Hand
X(mm)
2 4 6 8 10 12 14
60
80
100
120
140
Y(mm)
2 4 6 8 10 12 14
-1400
-1350
-1300
-1250
time (sec)
Z(mm)
2 4 6 8 10 12 14
250
300
350
400
450
Surgeon's Right Hand
X(mm)
2 4 6 8 10 12 14
-20
0
20
40
60
80
Y(mm)
2 4 6 8 10 12 14
-1400
-1350
-1300
-1250
time (sec)
Z(mm)
Shadow (1 s)
Characteristic Movement -> Holding
Estimation Function -> f1
Estimated Data
Real Data
Shadow (1 s)
Characteristic Movement -> Cut
Estimation Function -> f2
2 4 6 8 10 12 14
250
300
350
400
450
Surgeon's Left Hand
X(mm)
80
100
120
140
Y(mm)
25
30
35
40
45
X(mm)
2
4
6
8
Y(mm)
2 4 6 8 10 12 14
250
2 4 6 8 10 12 14
60
80
100
120
140
Y(mm)
2 4 6 8 10 12 14
-1400
-1350
-1300
-1250
time (sec)
Z(mm)
25
-2
2
4
6
8
Y(mm)
-140
-135
-130
-125
Z(mm)
2 4 6 8 10 12 14
250
300
350
X(
2 4 6 8 10 12 14
60
80
100
120
140
Y(mm)
-1350
-1300
-1250
Z(mm)
X(Y(mm)
-1
-1
-1
-1
Z(mm)

12. SystemEngineeringand
AutomationDepartment
María Belén Estebanez Campos
belen@uma.es
III. Implantation and Experiments
IMPLANTATION: SUTURE MANEUVER
INDEX
I. Overview
II.Pose
Estimation
Model
III.Experiments
IV.Conclusions
e2
Surgeon’s Left Hand Surgeon’s Right Hand
5 10 15 20 25 30 35 40 45
30
40
50
60
70
80
Y(mm)
5 10 15 20 25 30 35 40 45
-1400
-1350
-1300
-1250
time (sec)
Z(mm)
Z(mm)
5 10 15 20 25 30 35 40 45
5 10 15 20 25 30 35 40 45
time (sec)
5 10 15 20 25 30 35 4
-60
-40
-20
0
20
40
Y(mm)
5 10 15 20 25 30 35 4
-1400
-1350
-1300
-1250
time (sec)
Z(mm)
e1 e2 e3 e2 e4 e1 e4 e8 e7 e4e1

13. SystemEngineeringand
AutomationDepartment
María Belén Estebanez Campos
belen@uma.es
III. Implantation and Experiments
EXPERIMENTS: SUTURE MANEUVER
INDEX
I. Overview
II.Pose
Estimation
Model
III.Experiments
IV.Conclusions
Shadow (1 sec)
Characteristic Movement -> Holding
Estimation Function -> f1
5 10 15 20 25 30 35 40 45
300
350
400
450
Surgeon's Left Hand
X(mm)
5 10 15 20 25 30 35 40 45
30
40
50
60
70
80
Y(mm)
5 10 15 20 25 30 35 40 45
-1400
-1350
-1300
-1250
time (sec)
Z(mm)
5 10 15 20 25 30 35 40 45
250
300
350
400
450
Surgeon's Right Hand
X(mm)
5 10 15 20 25 30 35 40 45
-60
-40
-20
0
20
40
Y(mm)
5 10 15 20 25 30 35 40 45
-1400
-1350
-1300
-1250
time (sec)
Z(mm)
5 10 15 20 25 30 35 40 45
300
350
400
450
Surgeon's Left Hand
X(mm)
5 10 15 20 25 30 35 40 45
30
40
50
60
70
80
Y(mm)
5 10 15 20 25 30 35 40 45
-1400
-1350
-1300
-1250
time (sec)
Z(mm)
5 10 15 20 25 30 35 40 45
250
300
350
400
450
Surgeon's Right Hand
X(mm)
5 10 15 20 25 30 35 40 45
-60
-40
-20
0
20
40
Y(mm)
5 10 15 20 25 30 35 40 45
-1400
-1350
-1300
-1250
time (sec)
Z(mm)
5 10 15 20 25 30 35 40 45
300
350
400
450
Surgeon's Left Hand
X(mm)
5 10 15 20 25 30 35 40 45
30
40
50
60
70
80
Y(mm)
5 10 15 20 25 30 35 40 45
-1400
-1350
-1300
-1250
time (sec)
Z(mm)
5 10 15 20 25 30 35 40 45
250
300
350
400
450
Surgeon's Right Hand
X(mm)
5 10 15 20 25 30 35 40 45
-60
-40
-20
0
20
40
Y(mm)
5 10 15 20 25 30 35 40 45
-1400
-1350
-1300
-1250
time (sec)
Z(mm)
5 10 15 20 25 30 35 40 45
300
350
400
450
Surgeon's Left Hand
X(mm)
5 10 15 20 25 30 35 40 45
30
40
50
60
70
80
Y(mm)
5 10 15 20 25 30 35 40 45
-1400
-1350
-1300
-1250
time (sec)
Z(mm)
5 10 15 20 25 30 35 40 45
250
300
350
400
450
Surgeon's Right Hand
X(mm)
5 10 15 20 25 30 35 40 45
-60
-40
-20
0
20
40
Y(mm)
5 10 15 20 25 30 35 40 45
-1400
-1350
-1300
-1250
time (sec)
Z(mm)
Estimated Data
Real Data
Shadow (1 sec)
Characteristic Movement -> Turn around
Estimation Function -> f3
20 25 30 35 40 45
eon's Left Hand
20 25 30 35 40 45
5 10 15 20 25
250
300
350
400
450
Surgeon's Rig
X(mm)
5 10 15 20 25
-60
-40
-20
0
20
40
Y(mm)
20 25 30 35 40 45
20 25 30 35 40 45
5 10 15 20 25
250
300
350
400
X(mm)
5 10 15 20 25
-60
-40
-20
0
20
40
Y(mm)
-1300
-1250
mm)
0 25 30 35 40 45
0 25 30 35 40 45
5 10 15 20 25
-60
-40
-20
0
20
Y(mm)
5 10 15 20 25
-1400
-1350
-1300
-1250
Z(mm)
5 10 15 20 25 30 35 40 45
300
350
400
450
Surgeon's Left Hand
X(mm)
5 10 15 20 25 30 35 40 45
30
40
50
60
70
80
Y(mm)
-1250
250
300
350
400
450
X(mm)
-60
-40
-20
0
20
40
Y(mm)
5 10 15 20 25 30 35 40 45
300
350
400
450
Surgeon's Left Hand
X(mm)
5 10 15 20 25 30 35 40 45
30
40
50
60
70
80
Y(mm)
5 10 15 20 25 30 35 40 45
-1400
-1350
-1300
-1250
time (sec)
Z(mm)
5 10 15 20 25 30 35
250
300
350
400
450
Surgeon's Right Hand
X(mm)
5 10 15 20 25 30 35
-60
-40
-20
0
20
40
Y(mm) 5 10 15 20 25 30 35
-1400
-1350
-1300
-1250
time (sec)
Z(mm)
5 10 15 20 25 30 35 40 45
300
350
400
X(mm
5 10 15 20 25 30 35 40 45
30
40
50
60
70
80
Y(mm)
5 10 15 20 25 30 35 40 45
-1400
-1350
-1300
-1250
time (sec)
Z(mm)
250
300
350
400
X(mm
-60
-40
-20
0
20
40
Y(mm)
-1400
-1350
-1300
-1250
Z(mm)

14. SystemEngineeringand
AutomationDepartment
María Belén Estebanez Campos
belen@uma.es
CONCLUSIONS
To add a learning system so the MEKF may change its set of characteristic
movements depending on the surgeon
Improve the functions that model the characteristic movements or automate
their identification (splines, fourier series…)
FUTURE WORKS
A methodology for estimating the location of the surgeon’s tool has been
developed when there are occluded areas.
This work is valid when the occluded areas are produced in the short term
during the characteristic movements of the surgical tools or in presence of
obstacles.
The correct estimation can be made only when the maneuver is predicted
accordingly.
IV. Conclusions and Future Works
INDEX
I. Overview
II.Pose
Estimation
Model
III.Experiments
IV.Conclusions

15. SystemEngineeringand
AutomationDepartment
María Belén Estebanez Campos
belen@uma.es
Thanks for your attention
Surgical Tools Pose Estimation for a Multimodal HMI