This document summarizes a research project on developing an augmented reality (AR) based human-machine interface (HMI) for hydraulic excavators. The project aims to identify how AR can augment operator cognition and improve HMI designs. Technical solutions proposed include using heads-up displays and haptic feedback devices. Future work involves usability testing of AR-simulated HMIs and ergonomic analysis of head-up display inspired designs. The presenter learned about fluid power systems, AR, ergonomics and time management through this research experience.
Mattingly "AI & Prompt Design: The Basics of Prompt Design"
Research 2012a
1. Research Project Number
Hydraulic Excavator HMI DESIGN
USING Augmented Reality
Georgia Institute of Technology | Milwaukee School of Engineering | North Carolina A&T State University | Purdue
University
University of Illinois, Urbana-Champaign | University of Minnesota | Vanderbilt University
Presenter: Michael Ekundayo
Advisor: Dr. Eui Park
Graduate Supervisor: Joseph Akyeampong
North Carolina A&T State University
CCEFP REU 2012
June 4 – August 10, 2012
2. Outline
1. Background
2. Research Objective
3. Technical Solutions
4. Implementation
5. Lessons Learned
6. Idea
7. Future Work
8. Q&A
Event 2
Event Date(s)
3. Background – Fluid Power Systems
Hydraulics Pneumatics
Car lift Air
Compressor
Jet engine Wind
wiring turbines
Event 3
Event Date(s)
4. Background – Hydraulic
Excavators
• Work Domains include:
• Construction
(commercial and
residential)
• Mining
• Water & Sewer
• Forestry & Agriculture
Event 4
Event Date(s)
5. Background – Excavator
HMI
Issues
– Ease of Use (Cognitive)
• Less Intuitive
• Information deficient
– Safety (Physical)
• Layout of
Displays/Controls
• Repetitive use of
controls
Event 5
Event Date(s)
6. Background – Virtual vs.
Augmented Reality
Virtual Reality Augmented Reality
• A computer generated, • A combination of the a
interactive, three- real scene and a virtual
dimensional environment scene generated by a
in which a person is computer that augments
immersed. the scene with additional
information.
Event 6
Event Date(s)
7. Background - Augmented Reality
Technologies
An AR system consists of:
– a camera for acquiring video
images of the real world.
– a computer for generating virtual
objects.
– display unit (e.g. monitor/head-
mounted/projector) for merging
the virtual and real scenes.
– a tracking unit (e.g.
mechanical/optical/inertia),
trackers) for obtaining real-time
position/orientation of the user.
– fiducial markers for registering
(or overlaying) virtual objects on
the real world
– interaction devices (e.g.
joystick/mouse/haptic etc.) to
manipulate/navigate the
augmented environment.
Event 7
Event Date(s)
8. Examples of AR
Applications
• SkinVaders is the first
game to use Augmented
Reality to physically bring
a person body into a game.
• Google glasses developed
for A.R. application
Event 8
Event Date(s)
9. Examples of AR
Applications
IKEA uses augment
reality to fit a customer
with a product by imaging
the room with the
products like furniture,
refrigerators, dishwasher
and other products
Event 9
Event Date(s)
10. Research Objective
To develop an excavator HMI
utilizing augmented reality (AR)
technology
Event 10
Event Date(s)
11. Role of AR in Hydraulic
Excavator HMI Design
• The role of AR in hydraulic excavator
HMI design is to identify additional
information that can be used to
augment the operators cognition to
enhance his/her task performance.
• AR design strategies will be used to
recommend improved HMI designs
for hydraulic excavators.
Event 11
Event Date(s)
12. Technical Solutions (Ease of
Use)
Heads Up Display • Originated in jet fighters to provide
flight information to pilots without
requiring them to look away from their
usual viewpoints.
• sought to ease information
overload and to make the
instrumentation less complicated
• Common in aircrafts; being
introduced in cars
Hydraulic excavators will provide
great application platform to ease
the operators information
overload
Event 12
Event Date(s)
13. Approach/Technical
Solutions
• Haptic technology is a technology
Haptic Technology which takes advantage of the sense
of touch by applying forces,
vibrations, or motions to the user.
• Haptic devices (or haptic
interfaces) are mechanical devices
that mediate communication
between the user and the computer
• For hydraulic excavators it will
provide an additional modality
(sensory cue) to facilitate the
operators cognition, thereby
leading to enhanced usability.
Event 13
Event Date(s)
15. What I Learned
I have learned about
• Fluid Power Systems
• Hydraulic Excavators
• Augmented Reality
• Heads Up Displays
• Haptic Devices
• Ergonomics
• Time-management
• Readiness: to be prepared for everything
and anything
Event 15
Event Date(s)
16. Idea
• Google Earth has been explored for AR/MR related
applications to give remote viewing of geo-spatial
information (Frohlich et al., 2006) and urban planning
(Phan & Choo, 2010).
• Google Earth can be used with AR to guide the design
and implementation of HMIs for hydraulic excavators.
Question to be Addressed:
– How to connect Google earth with A.R. in real time?
Event 16
Event Date(s)
17. Future Work
• Assist with conducting usability testing
with AR-simulated HMIs
• Assist with performing ergonomics
analysis of HUD-inspired HMI in Jack.
Event 17
Event Date(s)