Social Robotics normally assumes visual feedback between robotic trainees and human trainers. Given that robots rarely have adequate visual perception/recognition, such systems are noisy and prone to judgment errors. One way to resolve this problem is to simplify the communication channel between humans and robots. This paper uses simple gravity+motion XYZV sensors ubiquitous in modern personal devices -- smartphones in particular -- to power gait-based exo-systems (power leg assist, etc). This paper discusses current work in progress on this topic, specifically (1) gait modeling and recognition of kick-in moments for hardware, (2) use of the XYZV channel in both directions, allowing humans to send feedback in realtime, (3) social robotics constructs and methods that support maximum flexibility in applications of the otherwise traditionally narrow-purpose hardware systems.

1. 2015/11/29
Marat Zhanikeev
maratishe@gmail.com
HCI研@別府
PDF: bit.do/151129
Towards Social Robotics on Smartphones
with Simple XYZV Sensor Feedback

2. (dumb) Robotics Today
• most robots today are dumb
• if you have all the necessary inputs, no room/
need for AI, just use your inputs
• the case of HAL: weight shifting,
bioelectricity, relative angles
M.Zhanikeev -- maratishe@gmail.com -- Towards Social Robotics on Smartphones with Simple XYZV Sensor Feedback -- bit.do/151129 2/19
2/19

3. From to Smart Robotics
• objective 1: there is a need for truely smart/flexible robotics
• objective 2: preferably within commodity reach (smartphones)
Increasing specialization
Increasingsmart-ness
Most
robots
today
Social
Robotics
M.Zhanikeev -- maratishe@gmail.com -- Towards Social Robotics on Smartphones with Simple XYZV Sensor Feedback -- bit.do/151129 3/19
3/19

4. Social Robotics Basics
• basic idea: start with a blank
robot and teach it a task
◦ humans are trainers
• optimization: reasoning by
reinforcement learning
• complexity: reduce
search space via human
feedback
M.Zhanikeev -- maratishe@gmail.com -- Towards Social Robotics on Smartphones with Simple XYZV Sensor Feedback -- bit.do/151129 4/19
4/19

5. Social Robotics in Soft- vs Hard-ware
• good parallel between soft- and hard-ware social robotics
• for higher reliability, vision/recognition in hardware robots should be
simplified
◦ reliable feedback gestures, possibly even without vision
• context is natural to software but is also important in hardware (more later)
Generic
Use
Teaching,
Guidance Reasoning
Human
Role
Hardware
Robots
Wide range of
behavior
Reinforcement
Learning
YES. Vision,
recognition
Guide only
Software
Automation
Any kind of
context
Bayesian
Classification
NO
Not needed
Guide and
decision -
maker
Potential for
simplification
M.Zhanikeev -- maratishe@gmail.com -- Towards Social Robotics on Smartphones with Simple XYZV Sensor Feedback -- bit.do/151129 5/19
5/19

6. Social Robotics in Context Management
• example of software robot that processes folksonomy bigdata
Robot
(careless)
Input
Human
Human
{structure}
(pinpoint)
Select
Browse
(or use otherwise)
Some
Knowledge
(folksonomies,
knowledge bases,
databases, indexes,
ontologies, etc.)
(metromaps )
guide, trainer
sources, crowd
Recommendation
Bot = Rebot
M.Zhanikeev -- maratishe@gmail.com -- Towards Social Robotics on Smartphones with Simple XYZV Sensor Feedback -- bit.do/151129 6/19
6/19

7. Social Robotics in Context (2)
0 5.2 10.4 15.6 20.8 26
Time/Learning Sequence
0
4
8
12
16
20
RebotOrdervsUserSelection
Linear fit (circle bullets)
Rebot: New Training
Rebot: Order of Selection in Rebot List
0 5.2 10.4 15.6 20.8 26
Time/Learning Sequence
0
4
8
12
16
20
RebotOrderofUserSelection
• starts off really dumn, but gets better with time
• diff between human and robot decisions decreases
• when diff is mostly zero, human trainer can go home
M.Zhanikeev -- maratishe@gmail.com -- Towards Social Robotics on Smartphones with Simple XYZV Sensor Feedback -- bit.do/151129 7/19
7/19

8. Social Robotics and Blackswans
• a useful "show me only the blackswans" application
• useful in : disaster scenarios, literature reviews, etc.
• crucial to remove human component as some point
Human
judgment
Auto
judgement
Folksonomy
M.Zhanikeev -- maratishe@gmail.com -- Towards Social Robotics on Smartphones with Simple XYZV Sensor Feedback -- bit.do/151129 8/19
8/19

9. Social Robotics and Blackswans (2)
Metromap
Classifier
Human
Check
Metromap
Fuzzy?
Cold?
Hot?
Robot (Automatic Classification)
Bad
Input
No
Yes
No
No
M.Zhanikeev -- maratishe@gmail.com -- Towards Social Robotics on Smartphones with Simple XYZV Sensor Feedback -- bit.do/151129 9/19
9/19

10. Simple XYZV Channel
M.Zhanikeev -- maratishe@gmail.com -- Towards Social Robotics on Smartphones with Simple XYZV Sensor Feedback -- bit.do/151129 10/19
10/19

11. Model of Robotic Decision Making
• on smartphones, easy access to XYZ gravity and V acceleration = XYZV input
• perfect if the same channel can be used for sensor input and human feedback
Model
Human
Robot
Actions
Constraints,
logic
Same -channel
interaction
Context
management
M.Zhanikeev -- maratishe@gmail.com -- Towards Social Robotics on Smartphones with Simple XYZV Sensor Feedback -- bit.do/151129 11/19
11/19

12. The Single-Channel Objective
Number of distinct data inputs
Reliability
Single-channel
Acceptable
level
1
Dumb curveSmart curve
M.Zhanikeev -- maratishe@gmail.com -- Towards Social Robotics on Smartphones with Simple XYZV Sensor Feedback -- bit.do/151129 12/19
12/19

13. Experiment (1)
• attach smartphone to one ankle
• detect up-the-stairs steps
• use human feedback to learn which
detections are wrong
M.Zhanikeev -- maratishe@gmail.com -- Towards Social Robotics on Smartphones with Simple XYZV Sensor Feedback -- bit.do/151129 13/19
13/19

14. Experiment (1) Feedback
1. smartphone notifies detected steps via
vibration
2. human punishes smartphone for bad
detections (negative feedback)
• stroke/hit your smarphone when it is wrong/
bad
• jerk/shake a leg to loosen it up
• both are natural interactions for humans
• positive feedback is harder (so far)
M.Zhanikeev -- maratishe@gmail.com -- Towards Social Robotics on Smartphones with Simple XYZV Sensor Feedback -- bit.do/151129 14/19
14/19

15. Experiment (1) : Results
negative mirror = moving average
“Stroke” feedback x
y
z
v
“Jerk” feedback
Event detected
Step
Stroke
Step
Jerk
M.Zhanikeev -- maratishe@gmail.com -- Towards Social Robotics on Smartphones with Simple XYZV Sensor Feedback -- bit.do/151129 15/19
15/19

16. Experiment (1) : Other Results
• about 100 experimental sessions
• 60% are FPs, 30% are FNs
• within the FPs, in 90% of them stroke/jerk feedback was detected
◦ 3s timeout, detect unusual XYZV signal right after vibration
◦ vibration itself is too small to register in XYZV
M.Zhanikeev -- maratishe@gmail.com -- Towards Social Robotics on Smartphones with Simple XYZV Sensor Feedback -- bit.do/151129 16/19
16/19

17. Experiment (2) Context
• current work in progress
• idea: establish context first, then detect steps
• experiment: walk with a smartphone in your
pocket
M.Zhanikeev -- maratishe@gmail.com -- Towards Social Robotics on Smartphones with Simple XYZV Sensor Feedback -- bit.do/151129 17/19
17/19

18. Experiment (2) Context (2)
x
y
z
v
flat walking
• horizontal/flat
walking
x
y
z
vwalking up the stairs
• walking up the
stairs
M.Zhanikeev -- maratishe@gmail.com -- Towards Social Robotics on Smartphones with Simple XYZV Sensor Feedback -- bit.do/151129 18/19
18/19

19. That’s all, thank you ...
M.Zhanikeev -- maratishe@gmail.com -- Towards Social Robotics on Smartphones with Simple XYZV Sensor Feedback -- bit.do/151129 19/19
19/19