PDR for lab-forming field and its benchmarking in ISMAR 2016 Workshop: Standards for Mixed and Augmented Reality

1. PDR for lab-forming field and
its benchmarking
Takeshi Kurata12, Ryosuke Ichikari1,
Tatsuro Myokan12, Masakatsu Kourogi1,
Katsuhiko Kaji3, and Nobuo Kawaguchi4
1AIST, 2Univ. of Tsukuba,
3Aichi institute of technology, 4Nagoya Univ.
ISMAR 2016 Workshop: Standards for Mixed and Augmented Reality (2016/9/23)

2. Lab-forming Field & Field-forming Lab
• Borrowing from “Terraforming”
• Lab-forming Field: Transforming a real field into a lab-like place.
• Field-forming Lab: Transforming a laboratory into a field-like place.
2

3. Big Data on Result/Behavior/Environment for
Lab-forming Field
3

4. Case study
in Japanese Restaurant “Ganko”
• Objectives
1. (for AIST) to test the CSQCC
(Computer-Supported QCC)
suites in a real service field.
2. (for the restaurant) to observe
effects of process
improvement planned by
CSQCC.
• Place
– Japanese cuisine restaurant
GANKO Ginza 4-chome
(Tokyo)
• Term
– 1st term
• January 12 to 18, 2011
– 2nd term
• February 3 to 9, 2011
4
Dining area Course dishes
1st term
(Jan. 12-18, 2011)
for observing
ordinary
operations
QC circle
for making
improvement
plans
2nd term
(Feb. 3-9, 2011)
for observing
improved
operations

5. 5 B2
B1
Dinning Area
Kitchen
Office room
Pantry

6. During Discussion in CSQCC
6
Trajectory of a wait staff in lunch time: 12:00-14:00
Fact: Going in and out of the kitchen/office to no small extent.
Possible result: Difficulty in concentrating on guest service.
Cause: Cell phone everywhere, but reservation book only in the office room.
Possible improvement: e-reservation book
Dinning Area
Kitchen
Office room
T. Fukuhara, R. Tenmoku, T. Okuma, R. Ueoka, M. Takehara, and T. Kurata, "Improving Service Processes based on
Visualization of Human-behavior and POS data: a Case Study in a Japanese Restaurant“, ICServ2013, pp.1-8.

7. PDR（Pedestrian Dead-Reckoning）
Estimates velocity vector, relative altitude, and action
type by measurements from a wearable sensor module.
 Wearing a sensor module on waist (2D SHS (Steps and Heading Systems) PDR)
 Easy to wear and maintain
 Easy to measure data for action recognition
 Relatively easily apply for handheld setting compared to shoe-mounted PDR
(3D-INS (Inertial Navigation System) PDR)
7
Handheld PDR
From PDR to PDRplus
10-axis sensors
• Accelerometers
• Magnetic sensors
• Gyro sensors
• Barometer
Shoe-mounted PDR
Waist-worn PDR

8. PDR Business Launched in Japan
Collaboration among ＮＴＴ Docomo, ZDC,
AKM, and AIST.
A PDR driven Indoor navigation mobile app
was released in Apr, 2015. (As of Sep, 2016,
440 areas including subways and
underground shopping arcades all across
Japan)
Collaboration between MegaChips and AIST.
Sensor Hub LSI “Frizz” was released in Jan,
2015 and MegaChips plans to sell 10 million
Frizz products in 2015. （Power-aware motion
coprocessor optimized for PDR)
8
• M. Kourogi and T. Kurata, Personal positioning based on walking locomotion analysis with
self-contained sensors and a wearable camera, ISMAR2003, pp. 103–112, (2003).

9. AR by PDR + Image registration
(1999-2003)
Panorama-based Annotation:
IWAR1999, ISWC2001,
ISMAR2003
G
Environmental map
A
B C D
E
A
B
C
F
Input frames
Position at which
a panorama is taken
Position
Direction
235 [deg]
5 [deg]
From the user’s
camera
Located Orientated
9

10. Frontier of PDR:
Walking direction estimation
10
• Tutorial: Personal Navigation with Handheld Devices by Valerie Renaudin, IPIN 2015.

11. Frontier of PDR:
Walking direction estimation
11
• Tutorial: Personal Navigation with Handheld Devices by Valerie Renaudin, IPIN 2015.
• Long Paper: Christophe Combettes, Valerie Renaudin, Comparison of Misalignment
Estimation Techniques Between Handheld Device and Walking Directions, IPIN 2015.
• FIS was proposed by Kourogi and Kurata in PLANS 2014.
“Globally, the FIS method provides better results
than the other two methods.”
Frequency analysis of Inertial Signals
Forward and Lateral Acc. Modeling
Principal Component Analysis

12. Overview: History of our PDR
12
ISWC2001
IWAR1999
ISMAR2003
PLANS2014
PLANS2010 ICServ2013
Docomo map navi
Image registration + Gyro
Panorama-based annotation (Image-registration-based positioning)
Image registration + PDR
PDRplus (PDR + Action recognition)
Handheld PDR
(Walking-direction estimation)
2015-
2015-
PDR module
2011-
Academia
Industry
Before PDR
ICAT2006 PDR + GPS + RFID

13. In the year of 2010
• iPhone 4: the first popular consumer mobile device
equipped with 9-axis sensors including accelerometers,
magnetic sensors, and gyro sensors
13
G-spatial EXPO 2010:
Handheld PDR on iPhone 4
(World’s first-ever live demo)
PLANS2010, PLANS2014

14. Global Trend on PDR
PDR R&D players have rapidly indicated their presence
all over the world on and after 2010.
Movea (France)
Sensor Platforms (USA)
CSR (UK)
TRX (USA)
Trusted Positioning (Canada)
14
Acquired by QualcommAcquired by InvenSenseAcquired by InvenSense
Acquired by Audience
Indoo.rs (USA)
SFO

15. Standardization on PDR Benchmarking
• PDR related R&D is highly active worldwide: Necessity for sharing
common measures.
• Description of the performance should be unified in spec sheets
and scientific papers.
• Different measures from absolute positioning methods such as
GNSS, Wi-Fi, and BLE are required for PDR, which is a method of
relative positioning.
• PDR Benchmark Standardization Committee was established in
2014 as a platform of the grassroots activity.
15
https://www.facebook.com/pdr.bms

16. Support Organizations
• Asahi Kasei Corporation, Asia Air Survey Co., Ltd. (Y. Minami), INTEC Inc.,
MTI Ltd., KDDI R&D Laboratories, Inc., KOKUSAI KOGYO CO., LTD.,
SHIBUYA KOGYO CO., LTD., Koozyt, Inc., SITESENSING, inc., Sharp
Corporation, Sugihara Software and Electron Industry Co., Ltd. (SSEI),
Information Services International-Dentsu, Ltd. (ISID), Hitachi, Ltd.,
Frameworx, Inc. (S. Watanabe), MULTISOUP CO.,LTD., Milldea, LLC,
Murata Manufacturing Co., Ltd., MegaChips Corporation, Recruit Lifestyle
Co., Ltd. (K. Ushida), RICOH COMPANY, LTD., Rei-Frontier Inc.,
• Aichi Institute of Technology (K. Kaji), Akashi National College of
Technology (I. Arai), Kanagawa Institute of Technology (H. Tanaka), Keio
University (S. Haruyama, N. Kohtake, M. Nakajima), University of Tsukuba
(T. Kurata), Tokyo Institute of Technology (S. Okada), Nagoya University
(N. Kawaguchi), Niigata University (H. Makino), Ritsumeikan University (N.
Nishio), National Institute of Advanced Industrial Science and Technology
(AIST) (T. Kurata, M. Kourogi), Human Activity Sensing Consortium
(HASC), Location Information Service Research Agency (LISRA)
• 33 organizations in Japan as of April, 2016
16

17. Activity Examples in Japan
17
1st meeting of the committee
(2014/05/22)
1st seminar in LBJ (2014/06/11)
2nd meeting of the committee
(2015/05/19)
2nd seminar in LBJ (2015/06/13)
Organized session in HCG
symposium (2014/12/17)
3rd seminar in LBJ (2016/06/10) LBJ: Location Business Japan
HCG: Human Communication Group
Organized session in HCG
symposium (2015/12/16)
3rd meeting of the committee
(2015/05/19)

19. Scene in data collection
19

20. Multi-Algorithm On-Site Evaluation System
• Evaluates the accuracy of each PDR algorithm automatically as often
as sensor data is uploaded to the server
• Provides trajectory images so that participants can compare their PDR
• algorithms in real time.
20
http://pdrsv.hasc.jp
K. Kaji, K. Kanagu, K. Murao, N. Nishio, K. Urano, H. Iida, N. Kawaguch, Multi-Algorithm On-
Site Evaluation System for PDR Challenge, ICMU2016, (to appear)

21. UbiComp/ISWC 2015 PDR Challenge Corpus
• Is now open to the public. (http://hub.hasc.jp/)
21
Routes 5
Devices 7
Subjects 93
# of pedestrian sensing data 241
# of pedestrian sensing data with
calibration data
230
# of pedestrian sensing data with
LIDAR data
10
Avg. of walking time [sec] 101
Avg. of moving distance [m] 115
Avg. of angular change [°] 606
K. Kaji, M. Abe, W. Wang, K. Hiroi, and N. Kawaguchi, UbiComp/ISWC 2015 PDR challenge
corpus, HASCA2016 (UbiComp2016 Proceedings: Adjunct), pp.696-704
Statistics of the corpus
Detailed route statistics of pedestrian
sensing data with calibration data

22. Open Data Contest in Logistics &
PDR Challenge in Warehouse
• Open data contest in logistics
by Frameworx
– Submission: 2016/4/18-2016/7/18
– Award ceremony: 2016/9/12
• PDR Challenge in Warehouse
– Now planning
– Will be held as a Japanese
domestic contest in 2016
– Will be held as an international
contest in IPIN 2017
22

23. PDR Challenge Series
• Ubicomp/ISWC 2015 PDR Challenge
– Scenario: Indoor Navigation
– On-site
– Continuous walking while keeping watching the navigation
screen by holding the smartphone
– Several minutes per trial
• 2016/2017 PDR Challenge in Warehouse (tentative)
– Scenario: Picking work in a warehouse
– Off-site
– Not only walking but various actions including picking and
carrying
– Several hours per trial
– Will be held in IPIN 2017
23

24. Examples of picking workers’
trajectories estimated by PDR + WMS
(Warehouse Management System)
24

25. How to design benchmark Indicators?
• Other aspects to be considered
– Reliability: Different measures from absolute positioning methods
are required for PDR
– Efficiency: Power consumption
– Repeatability: Temperature Hysteresis, Magnetic field, etc.
– Representativeness: How to hold, Route shape, etc.
25
Benchmark indicators of vision-based spatial registration and tracking for MAR
(ISO/IEC WD 18520)

26. How to compare and visualize?
26
Easy Difficult
Method 1
Easy Difficult
Method 2

27. How to compare and visualize?
27
Easy Difficult
Method1Method2

28. Competitions: IPIN and the others
(cf. EvAAL presentation in IPIN 2105 etc.)
28
IPIN year EvAAL, IPSN, UbiComp/ISWC
Zurich, Switzerland 2010 universAAL is launched
Guimaraes, Portugal 2011 EvAAL: indoor localization
Sidney, Australia 2012 EvAAL: + activity recognition
Montbeliard, France 2013 EvAAL: same as 2012
Busan, Korea
1st IPIN competition
2014
EvAAL: 3 floors, smartphone
IPSN: infrastruc. based + free
Banff, Canada
EvAAL-ETRI comp.
2015
EvAAL-ETRI: 6 floors, on/off-site
IPSN: infrastruc. based + free
UbiComp/ISWC: 2 floors, smartphone PDR,
90 subjects
Madrid, Spain
Indoor Localization
Competition
2016
IPIN: smartphone (on/off-site), PDR, Robot
IPSN: infrastruc. based + free, 2D/3D

29. IPIN2017
29
ISMAR2015

30. Thank you!
• AIST is now hiring for Tenure-track, Postdoc, and
RA (PhD) positions at Tsukuba, Japan.
• Target research fields are ↓ ↓ ↓
30