Astronauts and IoT: Toward True Human-Autonomy Teaming
•Transferir como PPTX, PDF•
1 gostou•388 visualizações
Dr. Alonso Vera, Chief, Human Systems Integration Division, NASA From the Nordic Digital Business Summit 2016 Helsinki, Finland 22 September 2016
Recomendados
Recomendados
Mais conteúdo relacionado
Mais procurados
Mais procurados (20)
Destaque
Destaque (15)
Semelhante a Astronauts and IoT: Toward True Human-Autonomy Teaming
Semelhante a Astronauts and IoT: Toward True Human-Autonomy Teaming (20)
Último
Último (20)
Astronauts and IoT: Toward True Human-Autonomy Teaming
- 1. Astronauts and IoT: Toward True Human-Autonomy Teaming Dr. Alonso Vera Chief, Human Systems Integration Division Nordic Digital Business Summit 2016 Helsinki, Finland 22 September 2016
- 2. Overview Three Themes - Not designing autonomous systems to interact with humans increases costs - The need ongoing need for human engagement with increasingly intelligent systems (IISs) - Both cognitive and physical - IoT allows IISs to reach into the physical world in an interactive way that has not been possible until now 10/19/2016 2
- 3. Machine Intelligence We appear to be at an exciting time with respect intelligent machines (again). • Four Related Areas of Development 1. Big Data - volume, velocity and variety 2. Deep Learning 3. Networked operations and cyber-physical systems 4. Moore’s Law (exponential growth, doubling of components on an integrated circuit every two years): faster, bigger computers driving change with increasing velocity • Stephen Hawking, Bill Gates and Elon Musk have all recently warned about the potential dangers of AI. • Also interesting time in terms of self-driving cars and companies with robotic operations/factories like Amazon, Tesla and Toyota • Big Blue, Watson, Pokerbot • Google DeepMind AI Division beats human at GO (Jan 2016) • First AI investment software hits Wall St. (Feb 2016)
- 4. Manpower Reduction: Start with the Human (Not the Technology) The Autonomy Paradox (Blackhurst, Gresham & Stone, 2011) • Autonomy doesn’t get rid of humans, it changes their roles • DoD has shifted from Levels-of- Automation to Cognitive Echelons As machine intelligence advances, the need for better human interfaces increases The Littoral Combat Ship Built to be operated by 45 sailors Dr. Larry Shattuck, NPS (pg. 13-15) http://human-factors.arc.nasa.gov/workshop/autonomy/download/presentations/Shaddock%20.pdf
- 5. Recent Developments The Littoral Combat Ship • New, highly autonomous vehicle In the end, the ship required 60 sailors, all E5 or above … and they are still encountering major issues
- 6. Apple Research on Teaming of IISs with Humans http://www.cultofmac.com/444551/why-apple-is-the-new-nasa/?&tc=em10/19/2016 6 • Characterizing calorie burn during swimming and using learning algorithms to tune the functionality to individual differences • Developed novel experimental hardware and tested on 700 swimmers • To develop a feature on one app for the new iWatch
- 7. Path to Collaborative, Human-in-the-Loop Planning Systems
- 8. NEEMO NASA Extreme Environments Mission Operations (NEEMO): Underwater laboratory off of the coast of Key Largo, Florida.
- 9. International Space Station CAST – Crew Autonomous Scheduling Test: • Playbook Check-Out by Scott Kelly on Station in August 2015. • Astronaut Peggy Whitson trained on Playbook in July 2016. • Astronaut self-scheduling study on ISS on Mission Increment 50/51 Nov 2016. ISS Mission Control • Three integrated planning systems: Power, Attitude Control and Crew Activity • Crew activity includes ESA JAXA and Payloads
- 10. What are the Challenges of Working with IISs? • Lack of transparency about intent, state awareness, risk/confidence posture, etc. • Part of the challenge is just the reverse though. Given that the IIS does not know what the human is trying to do, it is difficult for the system to know to engage in ways that are useful. 10/19/2016 10
- 13. IoT and IISs • IoT as a way for increasingly intelligent systems to have awareness of state in the physical world • Addressing the AI grounding problem • Beyond machine vision • Applied to manual human tasks • Assembly, maintenance, repair • Reduced training, errors • Task execution support • For complex tasks • For infrequently performed tasks 10/19/2016 13
- 14. Assembly, Maintenance and Repair on International Space Station 10/19/2016 14 The Rodent Habitat
- 15. How Can IoT Help? Challenge: loss of tools and loss of training • Tools and parts with embedded sensors • Data from sensors transmitted to the network by object • Objects know what they and can also capture and communicate other information about themselves • Location • Calibration status • History of failures, adjustments, repairs • Motion, torque, force, orientation, relative proximity May allow intelligent/adaptive procedure execution systems to guide humans though complex tasks with little or no training, as well as detect errors and reduce performance time 10/19/2016 15
- 16. Summer 2016 Student Intern Project
- 17. Internet of Things Microsoft Hololens Combine Augmented Reality and Internet-of-Things Mission AR-IOT
- 19. AR-IOT
- 20. Dividing procedure execution research into two parts Mission AR-IOT Task ProcedurePath Visualization
- 23. AR-IOT Prototype
- 24. Storyboards AR-IOT Voice command opens the selection menu. Voice or gesture selects the tools option User selects all wanted tools. Feedback occurs for each selection. Once ready, user selects “Find” to begin path. 1 2 3
- 25. Storyboards AR-IOT Menu begins to guide user. Simple HUD cues user to the destination. Spaced arrows lead user to location. HUD allows user access to a menu to cancel or change a path. Location has hologram of item over the storage compartment. 4 5 6
- 26. Storyboards AR-IOT Multiple of similar tools in a compartment with a hologram over the correct one. HUD changes to icons of the tools. Tool is secured in the compartment and the user picks it up. Haptic feedback and change of HUD when correct tool is located. Progress indicator fills in the current tool. 7 8 9
- 27. Storyboards AR-IOT When tool is located the path changes to begin guide to next location. HUD displays new location goal. Destination HUD changes to icon HUD. Line leads directly to the desired tool. A holographic overlay is also placed over the specific tool. 10 11 12
- 28. Storyboards AR-IOT Haptic and another change of HUD signal found object. Path changes to lead user to final destination. HUD also indicates location. Arrow or highlight of doorhandles indicates user to open the door (or module) 13 14 15
- 29. Storyboards AR-IOT 16 Once the user returns to the procedure location there will be indicators of places where they will need to install the tools based off of the IOT sensors and holograms. Once tools have been placed you can begin the procedure. The first step is to place the tools you've found in a secure bag. The user will see a holographic animation that shows you how to do this. 17 18
- 30. Step 2 is to attach your bag to the habitat. These animations are similar to the items own movement. 19 Storyboards AR-IOT 20 When the item is placed successfully there is a holographic blinking change as well as an update in your heads up display. Step 3. Open top access hatch. The animation will show the hatch opening and closing. 21
- 31. 22 Storyboards AR-IOT 23 Content such as videos or Skype calls can be displayed on the wall to provide support for the astronauts while they conduct procedures. Microsofts “Xray” feature can show objects containing other objects so users view isn't obstructed.
- 32. User Test 1B Recording
- 33. Insights & Opportunities AR-IOT Gestures (Select Tools) Many users said they preferred using voice over hand gesture but during user tests participants tended to use both. “Voice is more comfortable because I wasn’t sure about the depth of my gestures.” “Instead of navigation through menus there should be a way to simply say, let’s find the flashlight.”
- 34. Insights & Opportunities AR-IOT Guidance (Path) Some users felt that they didn’t need such an excessive guidance system which at times could feel unnecessary. “It would be condescending to an astronaut to tell them exactly where to go”
- 35. Insights & Opportunities AR-IOT Arrow Spacing (Path) Using calculated arrow spacing along side a linear path gave users an easy guidance system to locate their desired tool. “The arrows were positioned just right so I knew exactly where to turn” “Following the line was simple and intuitive”
- 36. Insights & Opportunities AR-IOT Destination (Locate Tool) Placing a hovering hologram over tools precise location proved to be a successful device to mark the destination. “I noticed the hologram and my immediate reaction was to go there.” “I saw a star, so it was pretty clear to me that something was there.”
- 37. Insights & Opportunities AR-IOT Precise Location (Locate Tool) A single line on the floor leading to a tools exact location was a successful method of marking the destination because the user could follow the line of tape with their eye to the exact destination. “You could clearly see that the highlighted drawer was where the item is”
- 38. Insights & Opportunities AR-IOT Visual Feedback (Locate Tool) Visual feedback after task completion was needed after the user found each tool to ensure they were on the right path. “I didn’t have any doubts that I have found the right flashlight but maybe if there was a dozen it would have been more helpful.” “Visual feedback would definitely be more helpful if there were different types of flashlights”
- 39. Insights & Opportunities AR-IOT Using Memory (Return) Many users were able to return to the procedure room while only using their memory and not relying on our guiding system. “To me it was pretty clear because I knew where this room was so I mostly tuned the path out”
- 41. AR-IOT Procedure Execution Animations & Procedure
- 42. AR-IOT Procedure Execution Animations & Procedure Holographic animation
- 43. Procedure Execution AR-IOT HUD We wanted an unintrusive heads up display that placed most of the information within the line itself.
- 45. From Cummings, M.L., "Man vs. Machine or Man + Machine?" IEEE Intelligent Systems, (2014) 29(5), p. 62-69. 2015
- 46. Architecture based on autonomy performing all skill and rule-based roles, as well as most knowledge-based roles. Manpower reduced by two orders of magnitude with remaining expert humans teaming with machine intelligence to solve complex problem solving under uncertainty. Machine intelligence for airspace management evolves from the outset to support teaming with small set of expert humans to support cooperative problem-solving. Adaptive 2035 Computers Humans e.g, current HITL for ATM Next-Gen research
- 47. Affordances from the Environment
- 48. Using Affordances • Application of Gibsons’ Ecological Psychology • Alternatives to using human central attention resource • A car more like a horse
- 49. Teaming of Human and Machine Intelligence • Even as computers get very “intelligent”, it is very likely that the nature of the their intelligence will be different than that of humans (unless they become omniscient or we program them to function just like humans) • Humans are particularly good at adaptive problem-solving and discovery, areas where there has been little machine intelligence progress • Successful efforts going forward will be those that wrap new machine intelligence capabilities around human competencies in order to get the most out of each Goal: Design the human into the process. Focus on how the system will communicate it’s state to the human so that the human can help in un-anticipated situations, and vice versa. What data and how it is presented to each agent such that each can bring its unique capabilities to bear on it.
- 50. The Economics of Human-Centered Automation • For lower costs, higher efficiencies and overall improved system performance: • Characterize nature of human roles (skills, rules, knowledge, expertise) and tasks (e.g., proportion of hard and soft constraints) • Wrap autonomy around remaining human roles from the beginning Dr. Jon Bornstein, DoD Autonomy Roadmap Autonomy Community of Interest http://www.defenseinnovationmarketplace.mil/resources/AutonomyCOI_NDIA_Briefing20150319.pdf Critical to shape the autonomy industry • e.g., Apple v. Littoral Combat Ship
- 51. Final Thoughts • Humans will remain important components of complex systems • Use human adaptive expertise as much as possible • Use human cognitive & perceptual system as much as possible in interactions • Robotics progressing faster than AI • Be aware of areas where you don’t have big data • Not all problems are associative in nature • Don’t assume search will solve all problems 10/19/2016 51
- 52. Thank you
Notas do Editor
- Very interested in speaking to this community because of the potential to shape the autonomy industry in a way that other sectors of the economy cannot. How to achieve what Apple has done for consumer products in the area of complex autonomous systems. It’s the last 5% that’s hardest – a lot of easy gains at first
- Notes from BBC report on GO “The rules are simpler than those of chess, but a player typically has a choice of 200 moves compared with about 20 in chess. There are more possible positions in Go than atoms in the universe, according to DeepMind's team. It can be very difficult to determine who is winning, and many of the top human players rely on instinct.” Three themes Robotics and human interfaces progressing quickly but real AI not so much Human intelligence will remain unique and of great value for decades Not designing autonomous systems to interact with humans increases costs When possible, use subcognitive channels for human-autonomy communication Petroleum industry The reason I was interested in speaking at this meeting is because I think the petroleum industry is particularly well placed to shift work in autonomy in the right direction. Note that I asked the person from the Navy who worked on the Littoral combat ship acquisition and he said the DoD, though relatively well funded cannot help shift things because of the ways in which contracts are worked by the government. Change is driven by consumers on an individual scale (e.g., resulting in Apple being one of the richest companies in the world) and by industry (as a customer).
- Using augmented reality and the Microsoft Hololens we can guide astronauts to their desired tools, as well as teach them how to conduct procedures while they're working. On the ISS tools are often lost and with IOT we can recognize which tool the astronaut has picked up. We can also tell when that tool or item is installed properly within the procedure.
- Using augmented reality and the Microsoft Hololens we can guide astronauts to their desired tools, as well as teach them how to conduct procedures while they're working. On the ISS tools are often lost and with IOT we can recognize which tool the astronaut has picked up. We can also tell when that tool or item is installed properly within the procedure.
- Our mission is to make this experience easier and save time by using emerging technologies not only for astronauts on the ISS but for future trips to mars.
- We chose to split this project up into two parts, one focusing on creating a guidance system to help astronauts to find their desired tool (Path Visualization). The other being a helper and training tool for astronauts to conduct procedures in space (Procedure Execution).
- During our last summer we used ESP8266 WiFi chips with ADXL345 accelerometers that communicate using I2C. Our algorithm used the jerk (derivative of acceleration) to detect when a hand-like motion had acted on the accelerometer, allowing the ESP chip to communicate with a server that acted as the procedure manager. The ESP chips were programmed using Arduino, a C based language
- At the center of the network of IoT devices, the server communicated with each chip using MQTT, a protocol specifically designed for IoT networks that allowed devices to publish and subscribe to “topics” for organization and expandable communication options. The server acted as a message “broker” that receives and forwards messages to the subscribed IoT devices.
- After we ran through our first prototype multiple times we felt we had enough data to make some minor changes and improve on our previous model. We decided to use a line guidance system that could not only lead a user through multiple rooms but also guide the eye to the precise location of the desired tool.
- We experimented with many ideas around holographic assistance such as arrow indicators or static holographic overlays. We landed on Holographic animations that would show you how to install an item as your doing it.
- Along with holographic animations we placed the procedure steps of the on the wall to assist and enable the astronauts to keep track of where they are. Once the steps are completed the animations change and the next three steps are shown. We wanted users to know they had installed the item in the correct way before moving on to the next step.
- We wanted an unintrusive heads up display that gave the user only one piece of information at a time. The majority of the information the user would need would be placed in the guidance line itself. Once the correct tool was picked up the HUD and Line would change.
- A car like a horse. Won’t walk off a cliff if you fall asleep but won’t get you home either
- Littoral combat ship and Apple, as examples