1. 1
Personal Context Discovery using
Unobtrusive Sensing
1st December 2012
CSI 2012
Arpan Pal
Principal Scientist and Research Head
Cyber physical Systems
Innovation Lab, Kolkata
Tata Consultancy Services (TCS)

2. 2
Human-in-Loop Cyber-physical Systems
Humans
Computing
Infrastructure
ICT Systems

3. 3
Human-in-Loop Cyber-physical Systems
Humans
Physical
Objects and
Infrastructure
Computing
Infrastructure
Cyber-physical Systems

4. 4
Human-in-Loop Cyber-physical Systems
Humans
Physical
Objects and
Infrastructure
Computing
Infrastructure
Human-in-loop Cyber-physical Systems

5. 5
Personal Context Discovery
Context - patterns of individual, group and societal behaviours.
Broadly classified into three categories –
Personal Physical Network Discovery
 Who is interacting with whom? What is the level of interaction? Who all are
part of similar-interest networks?
Individual Context Discovery
 Who is doing what?
 Who is thinking what?
Community Context Discovery
 Can we discover how a community / group behaves as a whole?

6. 6
Example Use Cases
Customer Behavior Study in Retail Stores
 Customer movement pattern
 Customer interaction pattern with shelves / merchandize
Crowdedness measure in public places
 Efficient scheduling of public transport
Wellness
 Activity and Work-out Quantification
 Pulse and other physiological parameter measurement
Organizational Behavior
 Team Efficiency / Best Practice Study
 Workspace Ergonomics - Stress Analysis
 People Profiling – Cognitive Load Analysis
Ref. - Alex Pentland et. al., MIT media Lab

7. 7
What do we need to Sense
 Location and Proximity
 Activity
 Identity
 Cognitive Load
 Physiological Parameters
Provide Personal
Context discovery as
a Service

8. 8
How to Sense
Requirement
• Needs to be Ubiquitous and Unobtrusive
• There should not be any new hardware / device to carry for an
individual
Approach
• Use smartphone-based sensors (GPS, accelerometer, compass,
microphone, camera)
• Use 3D surveillance cameras (like Kinect)
• Use wearable EEGs with mobile phone as gateway
• Augment with social network data and email data analytics
• Multimodal fusion of all the above
Issue
• Privacy can be an issue – needs to be handled on an use case-by-
use case basis
• Privacy vs. Utility

9. 9
Mobile Phone Based Sensing
Proximity / presence
– Using Bluetooth
– Using Wi-Fi
Location
– Using ultrasound beacon
– Using GPS (outdoors)
– Using Accelerometer / compass /
gyroscope
Activity
– Using Accelerometer
Interaction Level
– Using Microphone Audio
Identity
– From Network ID
Physiological Sensors
– Pulse rate using camera - PPG
On-board sensors
Accelerometer, GPS,
Compass, Gyroscope
Camera, Microphone
Network
Bluetooth, WiFi, 2G/GPRS, 3G
Network
2G/GPRS, Bluetooth
On-board sensors
Microphone, Camera

10. 10
Kinect Based Sensing
Human Identification
– Skeleton Model Based
– External Stimulus based refinement
Network Discovery
– Network discovery through proximity
– Level of Interaction through Audio
• 2D Camera with
IR depth sensor
• Excitation by IR
light pattern
• Directional Mic.
Human Interaction
– Activity Detection on 3D Point Cloud
– Physical object Identification
– Interaction with Objects
Example Activities
• People Discussion
• Give/Put/Take an object
• Enter/leave a room
• Handshaking

11. 11
EEG Based Sensing
Cognitive Load Detection
– Wearable EEG headset
– Biggest challenge is collecting annotated
data and removing movement artifacts
Signal Acquisition
Pre-processing
(Common Spatial
Pattern filter)
Feature Extraction
Classification /
Score generation
Measure of the cognitive
load
Feature-1
Feature-2
High cognitive load
Low cognitive load

12. 12
Soft sensing from Web
Unstructured Data
• Social network posts such as tweets,
facebook
• Blog posts
• Email bodies
Structured Data
• Social network profiles and network
information
• Email headers
• Tweet Attributes
Personal
Network

13. 13
Summary
Multimodal Fusion
Location Identity Activity
Sensors
• GPS, accelerometer, compass, microphone, camera on SmartPhone
• 2D / 3D surveillance cameras (like Kinect)
• Wearable / Mobile-phone hosted Physiological Sensors – Pulse, ECG, EEG
• Soft sensors from web and social network
Cognitive Load &
Phys. Sensing
Context Discovery
Physical, Individual & Community
Applications

14. Thank You
arpan.pal@tcs.com