Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...
Arpan pal csi2012
1. 1
Personal Context Discovery using
Unobtrusive Sensing
1st December 2012
CSI 2012
Arpan Pal
Principal Scientist and Research Head
Cyber physical Systems
Innovation Lab, Kolkata
Tata Consultancy Services (TCS)
5. 5
Personal Context Discovery
Context - patterns of individual, group and societal behaviours.
Broadly classified into three categories –
Personal Physical Network Discovery
Who is interacting with whom? What is the level of interaction? Who all are
part of similar-interest networks?
Individual Context Discovery
Who is doing what?
Who is thinking what?
Community Context Discovery
Can we discover how a community / group behaves as a whole?
6. 6
Example Use Cases
Customer Behavior Study in Retail Stores
Customer movement pattern
Customer interaction pattern with shelves / merchandize
Crowdedness measure in public places
Efficient scheduling of public transport
Wellness
Activity and Work-out Quantification
Pulse and other physiological parameter measurement
Organizational Behavior
Team Efficiency / Best Practice Study
Workspace Ergonomics - Stress Analysis
People Profiling – Cognitive Load Analysis
Ref. - Alex Pentland et. al., MIT media Lab
7. 7
What do we need to Sense
Location and Proximity
Activity
Identity
Cognitive Load
Physiological Parameters
Provide Personal
Context discovery as
a Service
8. 8
How to Sense
Requirement
• Needs to be Ubiquitous and Unobtrusive
• There should not be any new hardware / device to carry for an
individual
Approach
• Use smartphone-based sensors (GPS, accelerometer, compass,
microphone, camera)
• Use 3D surveillance cameras (like Kinect)
• Use wearable EEGs with mobile phone as gateway
• Augment with social network data and email data analytics
• Multimodal fusion of all the above
Issue
• Privacy can be an issue – needs to be handled on an use case-by-
use case basis
• Privacy vs. Utility
9. 9
Mobile Phone Based Sensing
Proximity / presence
– Using Bluetooth
– Using Wi-Fi
Location
– Using ultrasound beacon
– Using GPS (outdoors)
– Using Accelerometer / compass /
gyroscope
Activity
– Using Accelerometer
Interaction Level
– Using Microphone Audio
Identity
– From Network ID
Physiological Sensors
– Pulse rate using camera - PPG
On-board sensors
Accelerometer, GPS,
Compass, Gyroscope
Camera, Microphone
Network
Bluetooth, WiFi, 2G/GPRS, 3G
Network
2G/GPRS, Bluetooth
On-board sensors
Microphone, Camera
10. 10
Kinect Based Sensing
Human Identification
– Skeleton Model Based
– External Stimulus based refinement
Network Discovery
– Network discovery through proximity
– Level of Interaction through Audio
• 2D Camera with
IR depth sensor
• Excitation by IR
light pattern
• Directional Mic.
Human Interaction
– Activity Detection on 3D Point Cloud
– Physical object Identification
– Interaction with Objects
Example Activities
• People Discussion
• Give/Put/Take an object
• Enter/leave a room
• Handshaking
11. 11
EEG Based Sensing
Cognitive Load Detection
– Wearable EEG headset
– Biggest challenge is collecting annotated
data and removing movement artifacts
Signal Acquisition
Pre-processing
(Common Spatial
Pattern filter)
Feature Extraction
Classification /
Score generation
Measure of the cognitive
load
Feature-1
Feature-2
High cognitive load
Low cognitive load
12. 12
Soft sensing from Web
Unstructured Data
• Social network posts such as tweets,
facebook
• Blog posts
• Email bodies
Structured Data
• Social network profiles and network
information
• Email headers
• Tweet Attributes
Personal
Network
13. 13
Summary
Multimodal Fusion
Location Identity Activity
Sensors
• GPS, accelerometer, compass, microphone, camera on SmartPhone
• 2D / 3D surveillance cameras (like Kinect)
• Wearable / Mobile-phone hosted Physiological Sensors – Pulse, ECG, EEG
• Soft sensors from web and social network
Cognitive Load &
Phys. Sensing
Context Discovery
Physical, Individual & Community
Applications