Astute symposiume 10/10/2013 - Smart sensors userstate

1. A PROBABILISTIC MODEL FOR
COGNITIVE-AFFECTIVE USER STATE
AWARENESS
ARTEMIS JOINT UNDERTAKING
GRANT AGREEMENT: 269334
PROPOSAL ACRONYM:
ASTUTE

2. INTRODUCTION
Higher
data flow
WHY?
Data
Complex
services
Actions
Embedded Systems
are becoming more
and more complex
Different
applications
Astute project
Error
Information
Support Advice
Confirm
2

3. INTRODUCTION
ASTUTE project aims to improve usability of embedded systems by using
user state and context situation capture to provide pro-active decision
support via multi-modal interfaces.
CONTEXT
CAPTURE
CONTEXT
MODELING
ENGINE
PROACTIVE
DECISION
ENGINE
MULTI-MODAL
HMI
Pro-active decision support system based on human centered design able to
support user intentions while keeping him in control.
Astute project
3

4. USER AWARENESS
PHYSICAL
ENVIRONMENT
CONDITIONS
•Information
Priority
PROCESSING
DATA INPUTS
PHYSIOLOGIC
MEASURES
Astute project
•Decision
Support
USER
STATE
•Data delivery
•Keep in
control
4

5. USER STATE
PHYSICAL
ENVIRONMENT
CONDITIONS
PHYSIOLOGIC
MEASURES
ECG
SENSOR NETWORKS
USER
FEEDBACK
EXTERNAL DATABASES
VISUAL
ODOMETRY
SPECIFIC
MEASUREMENTS
Astute project
MOTION
ARTIFICIAL
VISION
EEG
BIOMETRICS
USER
STATE
5

6. USER STATE
EEG:
Electroencephalography is the measurement of electrical activity resulting from ionic
currents flows in brain neurons using multiple electrodes placed on the scalp.
It is commonly used in medicine for diagnostic applications, like epilepsy,
encephalopathies or sleep disorders, by analyzing its spectral content.
Further applications include EEG average analysis for cognitive sciences by analyzing
response to time-locked events and stimulus.
STRESS
FATIGUE
Astute project
RELAXATION
FATIGUE
CONCENTRATION
DISEASES
6

7. USER STATE
ECG:
Electrocardiography is the transthoracic measurement of electrical activity in heart
using different electrodes attached to user’s skin.
It is commonly used in medicine to measure heartbeats rate, size and position of
the different heart chambers and any effect of external source on heart.
Although ECG information is limited to physiological user status, it is useful to
complement EEG data to complement obtained information increasing system
performance and reliability.
How this information
is merged?
Astute project
7

8. RATIONALE & MOTIVATION
• Cognitive-affective states are relevant in the realisation of tasks that:
– Manage a large volume of information in the interface/system/process –
are a cognitive challenge
– Are critical (urgency, safety, health, sports)
– Involve people (human resources, leadership, coaching, social and
personal relationships)
• Availability of information on user state facilitates
interface/system/process pro-activeness, which is accomplished via
decision support built on top of data-/knowledge-based models,
providing:
– Alarms, feedback
– Recommendations for adapting the interface/system/process
Astute project
8

9. RATIONALE & MOTIVATION
• Present work provides a description of research and development of
a user state diagnostic system, within the broader context of the
European Artemis ASTUTE project
ASTUTE aims to develop advanced and innovative pro-active HMI
supported by reasoning engine system, for improving the way the
human being deals with complex and huge information quantities in
different operative conditions and contexts.
• A number of previous projects generally used a limited range of
sensors network, mainly focused on autonomous psychophysiological information, and used concrete context scenarios
• More technical effort may be done to incorporate measures of brain
activity, and thus to delineate a full picture of brain-body reaction
• A probabilistic model has been developed, given its capability to
handle uncertainty in sources information and inference, and sound
mathematical framework.
Astute project
9

10. OVERVIEW
Context
User Profile
Web
services
User state
Sensors
Mobile
Astute project
Mobile Cloud App
Cloud App
10

11. DATA CAPTURE
Frequency bands
α, β, θ,...
Calibration &
normalisation
Heart rate
EEG raw data
Calibration &
normalisation
To user
state model
User’s profile
Web services connectivity
Astute project
11

12. OUR COGNITIVE MODEL
• Three user states that are relevant in working environments and safetycritical tasks:
– stress, mental workload and fatigue
• A fourth state, namely inaptitude, is derived as a combination of the three
aforementioned user-states (stress, mental workload and fatigue):
– this is input to decision support, where recommendation for assistance will be
given
Astute project
12

13. OUR COGNITIVE MODEL
• Input used for diagnosis of user state comes from
– brain activity (EEG) and heart activity sensors
• This is complemented by selected
– predictive factors from context (context complexity, task workload) and
– user profile (experience, age, fitness),
– extracted from each user-case
Astute project
13

14. OUR COGNITIVE MODEL
As the number of parents of a node increase,
conditional probability tables (CPT) become larger.
Limit relations between nodes
if there’s conditional independence
Use canonical distributions
NoisyOR is used in our model
Adapt distributions with statistical analysis
of a large amount of training data
p(Symptom2 | Disease1, Disease2)
Symptom2
Disease1
False
True
False
False
80 %
20 %
False
True
40 %
60 %
True
False
30 %
70 %
True
Astute project
Disease2
True
20 %
80 %
14

15. OUR COGNITIVE MODEL
• This first model can
be extended with
“contrasts”
– for verifying the need
for assistance via the
state inaptitude.
– In the end, this
strategy improves the
decision reliability
Astute project
15

16. OUR COGNITIVE MODEL
• Assistance
recommendation is
provided
– alarms, voice, warning
messages, etc.
– assistance is based on
the expected utility in
order to advice the user
Astute project
16

17. USAGE EXAMPLE
Astute project
17

18. USAGE EXAMPLE
Astute project
18

19. SCHEMATIC ARCHITECTURE
Astute project
19

20. FUTURE WORK
• additional sensors
– for example to complement existing information or to cope with the
unavailability of specific sensors in particular settings
• further input parameters extracted from existing sensors
• improved pre-processing and fine-tuning of features can enhance
robustness
• new user’s cognitive and affective states
• further exploitation of user’s state contrasts and assistance
• exploring a dynamic version for anticipation or prediction of the
user’s state
• adaptability to users via training procedures and to improve the
prediction capacity of the cognitive model
• mobility and user profiling and personalization are key to our
system and deserve significant attention
Astute project
20

21. CONCLUSIONS
• we’ve shown present and future work within the
framework of ASTUTE project
• emphasis is on providing an integrated solution to monitor
and adapt the user’s state to the task demands in complex
contexts
– to design and implement a probabilistic cognitive model that is
predicting the user’s state based on complex context use-cases
– a set of sensors is diagnosing such user’s states based of brain
and heart rate evidence
• this solution is partially overcome by other projects,
however we add value in increasing the set of sensors,
range of user’s states in real and intensive scenarios
Astute project
21

22. THANKS FOR
YOUR ATTENTION!
Astute project
22