The document discusses approaches and challenges related to using virtual and augmented reality in healthcare. It provides an introduction and motivation for using medical AR/VR. Some healthcare scenarios discussed include using VR for radiologists in diagnostic reading rooms, virtual colonoscopy, and augmented reality for surgery. Challenges addressed include building networks of healthcare professionals, linguistic barriers, and "automagic worshippers" who expect too much without understanding specifications. The document emphasizes the importance of evaluation and usability testing with medical professionals.
(ANVI) Koregaon Park Call Girls Just Call 7001035870 [ Cash on Delivery ] Pun...
Matera Summer School AI&XR 2023.pdf
1. Approaches and Challenges to Virtual
and Augmented Reality in Health Care
Joaquim Jorge
Anderson Maciel
INESC-ID & IST / U Lisboa
Catarina Moreira
UTS, Sydney
Nuno Figueiredo
Grupo Lusíadas, Lisboa Portugal
2. Joaquim Jorge
Instituto Superior Técnico
Universidade de Lisboa
Visualization and Multimodal Interfaces
@ INESC-ID Lisboa
http://web.ist.utl.pt/jorgej
ACM Distinguished Member
Editor Computers & Graphics
UNESCO Chair AI & XR
Research Interests:
Virtual & Augmented Reality,
Multimodal Interfaces,
Graphical Modeling
3. Catarina Moreira
University Technology Sydney
Associate Professor
XAMI & INESC-ID
Co-PI UNESCO Chair AI&XR
Co-PI XAVIER Project
Research Interests:
Explainable AI
Data Science
Virtual & Augmented Reality,
4. Anderson Maciel
Graphics and Interaction @INESC-ID
Instituto Superior Técnico
Universidade de Lisboa
Virtual Reality
Haptics
Interactive Computer Graphics,
Computer-Aided Healthcare,
Interactive Visualization
5. Nuno Figueiredo
MD, PhD, FRCS
Head of Surgical Center
Lusiadas Knowledge Center
www.lusiadas.pt
Research Interests:
Image Guided Surgery
Augmented Reality
Computer-Aided Healthcare,
8. Reality–Virtuality Continuum
Real Reality Immersive VR
S
e
m
i
-
i
m
m
e
r
s
i
v
e
V
R
D
i
m
i
n
i
s
h
e
d
R
e
a
l
i
t
y
S
p
a
t
i
a
l
A
R
S
e
e
-
T
h
r
o
u
g
h
A
R
A
u
g
m
e
n
t
e
d
V
i
r
t
u
a
l
i
t
y
Mixed Reality
Augmented Reality
Extended Reality
9. Challenges
Building a network of healthcare professionals
Identifying current practices
Linguistic Barrier
Big Picture but no Specs
“Automagic worshippers”
21. VR for Radiologists in the Reading Room
https://www.youtube.com/watch?v=bsNwV4BhJOw
22. VR 4 Radiologists in the Reading ROOM
Evaluation
Radiology Neuroradiology
Gynecology
Surgery Dental Implantology
Obstetrics
Medical Specialities
5 no previous experience in VR
48. Machines can make the right predictions, but
not for the right reasons!
49. Machines can make the right predictions, but
not for the right reasons!
50. Machines can make the right predictions, but
not for the right reasons!
51. Machines can make the right predictions, but
not for the right reasons!
52. Machines can make the right predictions, but
not for the right reasons!
This is no Atelectasis!
It is just Fat!
53. Machines can make the right predictions, but
not for the right reasons!
Sporta et al (2021), Benchmarking saliency methods for chest X-ray interpretation. Medrxiv:2021.02.28.21252634v2
54. Machines can make the right predictions, but
not for the right reasons!
Sporta et al (2021), Benchmarking saliency methods for chest X-ray interpretation. Medrxiv:2021.02.28.21252634v2
Findings:
- Saliency methods (machine) perform
worse when compared to humans
- Errors between machine vs human
were larger in pathologies that were
smaller in size
55. Predictive algorithms widely
used in healthcare exhibited
racial biases that prevented
minority societal groups from
receiving extra care.
https://www.scientificamerican.com/article/racial-bias-found-in-a-major-
health-care-risk-algorithm/
60. Eye Tracking
Collecting radiologists eye movements can open new research directions in terms
of deep learning approaches:
Study radiologists’ cognitive
maps
Investigate fatigue / cognitive
load
Investigate how radiologists trained in different countries
assess these images / Radiologists’ degree of expertise
61.
62. Eye Tracking – Estimating Expertise from Eye Patterns
Borys and Plechawska-W
ó
jcik (2017), Eye-Tracking metrics in perception and visual attention research. European Journal of Medical Technologies,
(3):11-23.
74. The Importance of Clinical Data
Interviews with radiologists suggest that
they cannot make accurate diagnosis
without knowing the patients’ clinical
data.
77. Do Radiologists Have a Pattern?
NORMAL (healthy subject) Congestive Heart Failure Pneumonia
First Gazes in Silence
Gazes when
Speaking
78. Do Radiologists Have a Pattern?
NORMAL (healthy subject) Congestive Heart Failure Pneumonia
First Gazes in Silence
Gazes when
Speaking
PATTERNS FOUND!
79. Thank you to all contributors
Mr. Chihcheng (Richard) Hsieh Mr. André Luís Mr. Diogo Alvito
PhD candidate at
Centre for Data Science @ QUT
MSc at
Instituto Superior Técnico
MSc at
Instituto Superior Técnico
95. Healthcare Scenarios
UNESCO Chair: Digital Anatomy
Université Paris Descartes
Unité de recherche en Développement, Imagerie et
Anatomie - Prof. Vincent Delmas et Dr. J.F Uhl
116. Motivation
Problems
Lack of specialized surgeons in
remote, low-income regions
Robotic Teleoperation not
currently an option for open
surgery
Task shifting/task sharing
in LMIC as a last resort–
high risks
117. How to get specialized critical care
where care is not available?
Specialized surgeon in
developed country
Live Link
General surgery
118. Three Streams
Spot camera feed of surgical site
Stream of vitals monitor
360o immersive live feed of
surgery room
122. Prototype Implementation
Surgery Room PC
Server
Surgery Room Monitor
VR Application
Surgeon side
Send live video feed
Send annotated video feed
360o camera
Spot Camera
Vitals data
Show annotated video
Send
recorded
videos
After
closing
the call
✓Interacts with video
✓Annotates videos
✓Video is Recorded
✓Audio
123. Prototype Implementation
• Surgeon annotates on surgical site video.
• Annotations are sent to local surgery team
• Each annotation is saved
• Two most recent are accessible
126. Evaluation
+ found to be highly usable
+ viewed as potentially beneficial
+ high sense of presence
– evaluation not (yet) done with MDs
– minor UX/design issues
133. Next steps
Short term
test tool with surgeons
improve surgery room interface
Long term
AR in surgery room with real-time fitting of annotations to real organ
Closing the telepresence loop by adding AR representation of remote
surgeon
144. Acknowledgments
VRRRRoom Maurício Sousa, Daniel Mendes
CAVE COLON Pedro Borges, Daniel Medeiros, João Serras
Voxel TIPs Rita Mendes, Pedro Parreira, Soraia Paulo
Anatomy Studio Maurício Sousa, Daniel Mendes, Rafael Anjos,
Soraia Paulo
Voxel Explorer Pedro Parreira, Soraia Paulo
TOOTHFAIRY Filipe Relvas, Soraia Paulo
Laparoscopic HUD Miguel Belo
ImplantAR Tiago Jerónimo
SURVIVRS Regis Kopper (UNC)
145. Special Thanks to
Ramesh Raskar, MIT
Anders Ynnerman, Linkoping U.
Maurício de Sousa, U Toronto
Daniel Pires de Sá Medeiros, U Glasgow
Daniel Tavares Mendes, U Porto
Rafael Kuffner dos Anjos, UCL, London, UK
Soraia Figueiredo Paulo, INESC-ID
Beatriz Peres, U Madeira, INESC-ID
160. Digital Anatomy: Applications of Virtual, Mixed and Augmented Reality
Editors: Uhl, J.-F., Jorge, J., Lopes, D.S., Campos, P.F.
Springer: Human-Computer Interaction Series, 2021
ISBN 978-3-030-61905-3
Laparoscopy with augmented reality adaptations
ER Zorzal, JMC Gomes, M Sousa, P Belchior, PG da Silva, N Figueiredo, DS Lopes, Joaquim Jorge, 2020
Journal of biomedical informatics 107, 103463
Camera Travel for Immersive Colonography
SF Paulo, D Medeiros, P Borges, J Jorge, DS Lopes
arXiv preprint arXiv:2010.07798 2020
Improving Camera Travel for Immersive Colonography
SF Paulo, D Medeiros, PB Borges, J Jorge, DS Lopes
2020 IEEE Conference on Virtual Reality and 3D User Interfaces
161. Soraia Figueiredo Paulo, Filipe Relvas, Hugo Nicolau, Yosra Rekik, Vanessa Machado, João Botelho, José
João Mendes, Laurent Grisoni, Joaquim Jorge, Daniel Simões Lopes, Touchless interaction with medical
images based on 3D hand cursors supported by single-foot input: A case study in dentistry, Journal of
Biomedical Informatics, Volume 100, 2019, 103316, DOI: 10.1016/j.jbi.2019.103316
Daniel Simões Lopes, Filipe Relvas, Soraia Paulo, Yosra Rekik, Laurent Grisoni, Joaquim Jorge. 2019.
FEETICHE: FEET Input for Contactless Hand gEsture Interaction. In The 17th International Conference on
Virtual Reality Continuum and its Applications in Industry (VRCAI ’19), November14–16, 2019, Brisbane,
QLD, Australia. ACM, New York, NY, USA, 10 pages. DOI: 10.1145/3359997.3365704
Ezequiel R. Zorzal, Maurício Sousa, Daniel Mendes, Rafael Kuffner dos Anjos, Daniel Medeiros, Soraia
Figueiredo Paulo, Pedro Rodrigues, José João Mendes, Vincent Delmas, Jean-Francois Uhl, José
Mogorrón, Joaquim Armando Jorge, Daniel Simões Lopes, Anatomy Studio: a Tool for Virtual Dissection
Through Augmented 3D Reconstruction, Computers & Graphics, DOI: 10.1016/j.cag.2019.09.006
162. D.S. Lopes, D. Medeiros, S.F. Paulo, P.B. Borges, V. Nunes, V. Mascarenhas, M. Veiga, J.A. Jorge,
Interaction Techniques for Immersive CT Colonography: A Professional Assessment, In: Frangi A., Schnabel
J., Davatzikos C., Alberola-López C., Fichtinger G. (eds) Medical Image Computing and Computer Assisted
Intervention – MICCAI 2018. MICCAI 2018. Lecture Notes in Computer Science, vol 11071, Pages 629–637,
Springer, Cham, 2018. DOI: 10.1007/978-3-030-00934-2_70
M. Sousa, D. Mendes, S. Paulo, N. Matela, J. Jorge, D.S. Lopes, VRRRRoom: Virtual Reality for radiologists
in the reading room, Proceedings of the 35th Annual ACM Conference on Human Factors in Computing
Systems (CHI 2017), New York: ACM Press, 2017. DOI: 10.1145/3025453.3025566
D.S. Lopes, P.F. Parreira, S.F. Paulo, V. Nunes, P.A. Rego, M.C. Neves, P.S. Rodrigues, J.A. Jorge, On the
utility of 3D hand cursors to explore medical volume datasets with a touchless interface, Journal of Biomedical
Informatics, 72, Pages 140–149, 2017. DOI: 10.1016/j.jbi.2017.07.009
Daniel Simões Lopes, Pedro F. Parreira, Ana R. Mendes, Vasco M. Pires, Soraia F. Paulo, Carlos Sousa,
Joaquim A. Jorge, Explicit design of transfer functions for volume-rendered images by combining histograms,
thumbnails, and sketch-based interaction, The Visual Computer, December 2018, Volume 34, Issue 12, pp
1713–1723. DOI: 10.1007/s00371-017-1448-8
163. R Tang, XD Yang, S Bateman, J Jorge, A Tang, Physio@ Home: Exploring visual guidance and feedback
techniques for physiotherapy exercises, ACM SIGCHI 2015
João Vieira, Maurício Sousa, Artur Arsénio and Joaquim Jorge SleeveAR: Augmented Reality for
Rehabilitation using Realtime Feedback, ACM IUI 2016
Beatriz Peres, Pedro F. Campos, Aida Azadegan, A Persuasive Approach in Using Visual Cues to Facilitate
Mobility Using Forearm Crutches. BCSS@PERSUASIVE 2019
Beatriz Peres, Pedro F. Campos, Aida Azadegan , Digitally Augmenting the Physical Ground Space with Timed
Visual Cues for Crutch-Assisted Walking. CHI Extended Abstracts 2019
Beatriz Peres, Pedro F. Campos, Aida Azadegan , A Digitally-Augmented Ground Space with Timed Visual
Cues for Facilitating Forearm Crutches’ Mobility, Full paper , Interact 2019
.