1. PBN: Towards Practical Activity
Recognition Using Smartphone-
based Body Sensor Networks
The real authors …Matt Keally, Gang Zhou, Guoliang Xing1, Jianxin Wu2, and
Andrew Pyles
College of William and Mary, 1Michigan State University, 2Nanyang Technological University
Sensys2011
Nakagawa Keijiro
The University of Tokyo, Sezaki Lab

2. Outline
! Related Work
! Proposal
! Abstract
! Experimentation
! The most important point
! AdaBoost
! Sensor Selection
! Results
! Future
2

3. Related Work
Body Sensor Networks
v Athletic Performance
v Health Care
v Activity Recognition
LAN

4. Related Work
! No mobile or on-body aggregator
! (Ganti, MobiSys ‘06), (Lorincz, SenSys ‘09), (Zappi, EWSN ‘08)
! Use of backend servers
! (Miluzzo, MobiSys ‘10), (Miluzzo, SenSys ‘08)
! Single sensor modality or separate classifier per modality
! (Azizyan, MobiCom ‘09), (Kim, SenSys ‘10), (Lu, SenSys ‘10)
! Do not provide online training
! (Wang, MobiSys ‘09), (Wachuri, UbiComp ‘10)

5. Proposal
PBN: Practical Body Networking

6. Abstract
Activity recognition
Hardware: TinyOS + Android
Software: Android Application
USB

7. Abstract
Sensor Node
Sample
Local Agg. Sensor
Sensor
Sensor Controller
Agg. Data Start/Stop
802.15.4
Base Station Node USB
Phone
Sample TinyOS
GUI Comm. Stack
Controller
Labeled Data Activity Decision, Request Ground Truth Agg. Data
Sensor
Sensor Ground Truth Selection
Sensor
Sensor Management
Training Data
Agg.
Data Retraining Activity Sensor
Local Agg. Classification
Detection Selection
Activity Prob, Agg. Data Input Sensors
7

8. Experimentation
v 2 subjects, 2 weeks
v Android Phone
3-axis accelerometer, WiFi/GPS Localization
v 5 IRIS Sensor Motes
2-axis accelerometer, light, temperature, acoustic, RSSI
Node ID Location
0 BS/Phone
1 L. Wrist
2 R. Wrist
3 L. Ankle
4 R. Ankle
5 Head

9. Abstract
Sensor Node
Sample
Local Agg. Sensor
Sensor
Sensor Controller
Agg. Data Start/Stop
802.15.4
Base Station Node USB
Phone
Sample TinyOS
GUI Comm. Stack
Controller
Labeled Data Activity Decision, Request Ground Truth Agg. Data
Sensor
Sensor Ground Truth Selection
Sensor
Sensor Management
Training Data
Agg.
Data Retraining Activity Sensor
Local Agg. Classification
Detection Selection
Activity Prob, Agg. Data Input Sensors
9

10. The most important point
AdaBoost
(Activity Classification)
Sensor Selection

11. AdaBoost
Ensemble Learning: AdaBoost.M2 (Freund, JCSS ‘97)
v Lightweight and accurate
v Maximizes training accuracy for all activities
v Many classifiers (GMM, HMM) are more demanding
Iteratively train an ensemble of weak classifiers
v Training observations are weighted by misclassifications
v At each iteration:
l Train Naïve Bayes classifiers for each sensor
l Choose the classifier with the least weighted error
l Update weighted observations
The ensemble makes decisions based on the weighted decisions of
each weak classifier

12. Sensor Selection
v Identify both helpful and redundant sensors
v Train fewer weak classifiers per AdaBoost iteration
v Bonus: use even fewer sensors
Raw Data Correlation
Sensors

13. Sensor Selection
Goal: determine the sensors that AdaBoost chooses using correlation
Find the correlation of each pair of sensors selected by AdaBoost
Use average correlation as a threshold for choosing sensors
2,MIC
2,MIC
Unused
3,TEMP 1,ACC 1,ACC
All Sensors AdaBoost
3,MIC 1,LIGHT
3,MIC
2,TEMP
3,TEMP
Selected
2,TEMP
1,LIGHT

14. Sensor Selection
Goal: determine the sensors that AdaBoost chooses using correlation
Find the correlation of each pair of sensors selected by AdaBoost
Use average correlation as a threshold for choosing sensors
2,MIC
2,MIC
Unused
3,TEMP 1,ACC 1,ACC
All Sensors AdaBoost
3,MIC 1,LIGHT
3,MIC
2,TEMP
3,TEMP
correlation(2,TEMP;
1,LIGHT)
Selected
2,TEMP
1,LIGHT

15. Sensor Selection
Goal: determine the sensors that AdaBoost chooses using correlation
Find the correlation of each pair of sensors selected by AdaBoost
Use average correlation as a threshold for choosing sensors
2,MIC
2,MIC
Unused
3,TEMP 1,ACC 1,ACC
All Sensors AdaBoost
3,MIC 1,LIGHT
3,MIC
2,TEMP
3,TEMP
correlation(2,TEMP;
1,LIGHT)
Selected
2,TEMP
correlation(3,MIC;
3,TEMP)
1,LIGHT

16. Sensor Selection
Goal: determine the sensors that AdaBoost chooses using correlation
Find the correlation of each pair of sensors selected by AdaBoost
Use average correlation as a threshold for choosing sensors
2,MIC
2,MIC
Unused
3,TEMP 1,ACC 1,ACC
All Sensors AdaBoost
3,MIC 1,LIGHT
3,MIC
2,TEMP
3,TEMP
correlation(2,TEMP;
1,LIGHT)
Selected
2,TEMP
correlation(3,MIC;
3,TEMP)
1,LIGHT
…
Set threshold α based on average correlation:α
=
μcorr
+
σcorr

17. Sensor Selection
Unused
2,MIC
3,TEMP 1,ACC
All Sensors
3,MIC 1,LIGHT
2,TEMP
Selected AdaBoost
correlation(1,ACC;
1,LIGHT)
≤
α

18. Sensor Selection
Unused
2,MIC
3,TEMP 1,ACC
All Sensors
3,MIC 1,LIGHT 1,ACC
2,TEMP
1,LIGHT
Selected AdaBoost
correlation(2,TEMP;
1,ACC)
>
α
acc(2,TEMP)
>
acc(1,ACC)

19. Sensor Selection
Unused
2,MIC
3,TEMP 1,ACC 1,ACC
All Sensors
3,MIC 1,LIGHT
2,TEMP
1,LIGHT
Selected AdaBoost
2,TEMP
correlation(1,ACC;
3,TEMP)
≤
α

20. Sensor Selection
Unused
2,MIC
3,TEMP 1,ACC 1,ACC
All Sensors
3,MIC 1,LIGHT 3,TEMP
2,TEMP
1,LIGHT
Selected AdaBoost
2,TEMP

21. Results
We improved excluding redundant sensors more about 15% than others by
“AdaBoost+SS”

22. Conclution
! PBN: Towards practical BSN daily activity recognition
! PBN provides:
! Strong classification performance
! Retraining detection to reduce invasiveness
! Identification of redundant resources
! Future Work
! Extensive usability study
! Improve phone energy usage

23. Thank you for your listening
23

24. Appendix

25. Kullback–Leibler divergence
確率論と情報理論における2つの確率分布の差異を計る尺度
P 、 Q を離散確率分布とするとき、P の Q に対するカルバック・ライブラー情報量は
以下のように定義される。
ここでP(i) 、Q(i) はそれぞれ確率分布P 、 Q に従って選ばれた値がi になる確率。

26. AdaBoost training Algorihm
Input: Max iterations T, training obs. vector Oj for
each sensor s j ∈ S, obs. ground truth labels
Output: Set of weak classifiers H
Initialize observation weights D1 to 1/n for all obs.
for t=1toT do
forsensorsj∈Sdo Train weak classifier ht,j using
obs. Oj, weights Dt Get weighted error εt, j for ht,
j using labels [8]
end for
Add the ht, j with least error εt to H by choosing
ht, j with least error εt Set Dt+1 using Dt ,
misclassifications made by ht [8]
endfor

27. Raw Correlation Threshold for
Sensor Selection
Input: Set of sensors S selected by AdaBoost, training ob-
servations for all sensors O, multiplier n
Output: Sensor selection threshold α
R = 0/ // set of correlation coefficients for all
combinations of sensors si and s j in S do
Compute correlation coefficient r = |rOi ,O j |
R=R∪{r} end for
// compute threshold as avg + (n * std. dev. ) of R α =
μR + nσR

28. Sensor Selection Using Raw
Correlation
Input: SetofallsensorsS,training observations for all sensors O,
threshold α
Output: Selected sensors S∗ to give as input to AdaBoost
S ∗ = 0/ E = 0/ // set of sensors we exclude for all
combinations of sensors si and s j in S do
Compute correlation coefficient r = |rOi ,O j | if r < α then
i f s i ∈/ E t h e n S ∗ = S ∗ ∪ { s i }
i f s j ∈/ E t h e n S ∗ = S ∗ ∪ { s j } else if r ≥ α and acc(si) >
acc(sj) then
// use accuracy to decide which to add to S∗ i f s i ∈/ E t h e n
S ∗ = S ∗ ∪ { s i } E = E ∪{sj}; S∗ = S∗ {sj}
else i f s j ∈/ E t h e n S ∗ = S ∗ ∪ { s j } E = E ∪{si}; S∗ = S∗
{si}
end if endfor

29. Personal Sensing Applications
! Body Sensor Networks
! Athletic Performance
! Health Care
! Activity Recognition
Heart Rate Monitor
Pulse Oximeter
Mobile Phone Aggregator
29

30. A Practical Solution to Activity Recognition
! Portable
! Entirely user controlled
! Computationally lightweight
! Accurate
On-Body Sensors Phone
+Sensing Accuracy +User Interface
+Energy Efficiency +Computational Power
+Additional Sensors
30

31. Challenges to Practical Activity Recognition
! User-friendly
! Hardware configuration
! Software configuration
! Accurate classification
! Classify difficult activities in the presence of dynamics
! Efficient classification
! Computation and energy efficiency
! Less reliance on ground truth
! Labeling sensor data is invasive
31

32. PBN: Practical Body Networking
! TinyOS-based motes + Android phone
! Lightweight activity recognition appropriate for motes and phones
! Retraining detection to reduce invasiveness
! Identify redundant sensors to reduce training costs
! Classify difficult activities with nearly 90% accuracy
32

33. Hardware: TinyOS + Android
! IRIS on-body motes, TelosB base station, G1 phone
! Enable USB host mode support in Android kernel
! Android device manager modifications
! TinyOS JNI compiled for Android
33

34. Software: Android Application
Sensor Configuration Runtime Control and Feedback Ground Truth Logging
34

35. Data Collection Setup
! Classify typical daily activities, postures, and environment
! Previous work (Lester, et. al.) identifies some activities as
hard to classify
! Classification Categories:
Environment Indoors, Outdoors
Posture Cycling, Lying Down, Sitting, Standing, Walking
Activity Cleaning, Cycling, Driving, Eating, Meeting,
Reading, Walking, Watching TV, Working
35

36. PBN Architecture
Sensor Node
Sample
Local Agg. Sensor
Sensor
Sensor Controller
Agg. Data Start/Stop
802.15.4
Base Station Node USB
Phone
Sample TinyOS
GUI Comm. Stack
Controller
Labeled Data Activity Decision, Request Ground Truth Agg. Data
Sensor
Sensor Ground Truth Selection
Sensor
Sensor Management
Training Data
Agg.
Data Retraining Activity Sensor
Local Agg. Classification
Detection Selection
Activity Prob, Agg. Data Input Sensors
36

37. Retraining Detection
! Body Sensor Network Dynamics affects accuracy during runtime:
! Changing physical location
! User biomechanics
! Variable sensor orientation
! Background noise
! How to detect that retraining is needed without asking for ground
truth?
! Constantly nagging the user for ground truth is annoying
! Perform with limited initial training data
! Maintain high accuracy
37

38. Retraining Detection
! Measure the discriminative power of each sensor: K-L divergence
! Quantify the difference between sensor reading distributions
Sensors
! Retraining detection with K-L divergence:
! Compare training data to runtime data for each sensor
38

39. Retraining Detection
! Training
! Compute “one vs. rest” K-L divergence for each sensor and activity
Walking Driving Working
Training Data Ground Truth:
For each sensor:
Sensors
1,LIGHT Walking Data
Partition:
1,ACC
2,MIC DKL(Twalking,Tother)
=
vs.
Walking {Driving, Working}
…
Training Data Distribution Training Data Distribution
39

40. Retraining Detection
! Runtime
! At each interval, sensors compare runtime data to training data
for current classified activity
Sensors Current AdaBoost Classified Activity: Walking
For each sensor:
1,LIGHT
1,ACC
DKL(Rwalking,Twalking)
=
vs.
2,MIC Walking Walking
Runtime Data Distribution Training Data Distribution
…
40

41. Retraining Detection
! Runtime
! At each interval, sensors compare runtime data to training data
for current classified activity
! Each individual sensor determines retraining is needed when:
DKL(Rwalking,Twalking)
>
DKL(Twalking,Tother)
Intra-activity divergence Inter-activity divergence
vs. vs.
Walking Walking Walking {Driving, Working}
Runtime Data Distribution Training Data Distribution Training Data Distribution Training Data Distribution
41

42. Retraining Detection
! Runtime
! At each interval, sensors compare runtime data to training data
for current classified activity
! Each individual sensor determines retraining is needed
! The ensemble retrains when a weighted majority of sensors
demand retraining
42

43. Ground Truth Management
! Retraining: How much new labeled data to collect?
! Capture changes in BSN dynamics
! Too much labeling is intrusive
! Balance number of observations per activity
! Loose balance hurts classification accuracy
! Restrictive balance prevents adding new data
! Balance multiplier
! Each activity has no more than δ times the average
! Balance enforcement: random replacement
43

44. Sensor Selection
! AdaBoost training can be computationally demanding
! Train a weak classifier for each sensor at each iteration
! > 100 iterations to achieve maximum accuracy
! Can we give only the most helpful sensors to AdaBoost?
! Identify both helpful and redundant sensors
! Train fewer weak classifiers per AdaBoost iteration
! Bonus: use even fewer sensors
44

45. Sensor Selection
Choosing sensors with slight correlation yields the highest accuracy
45

46. Evaluation Setup
! Classify typical daily activities, postures, and environment
! 2 subjects over 2 weeks
! Classification Categories:
Environment Indoors, Outdoors
Posture Cycling, Lying Down, Sitting, Standing, Walking
Activity Cleaning, Cycling, Driving, Eating, Meeting,
Reading, Walking, Watching TV, Working
46

47. Classification Performance
47

48. Classification Performance
48

49. Classification Performance
49

50. Retraining Performance
50

51. Sensor Selection Performance
51

52. Application Performance
! Power Benchmarks
! Training Overhead
! Battery Life
Mode CPU Memory Power
Idle (No PBN) <1% 4.30MB 360.59mW
Sampling (WiFi) 19% 8.16MB 517.74mW
Sampling (GPS) 21% 8.47MB 711.74mW
Sampling (Wifi) + Train 100% 9.48MB 601.02mW
Sampling (WiFi) + Classify 21% 9.45MB 513.57mW
52