A recent talk about designing, building, and analysing the data from Emotion Sense and Q Sense -- Android apps for health monitoring and intervention.

1. Opportunities and Challenges of
Using Smartphones for Health
Monitoring and Intervention
@neal_lathia
Computer Laboratory
University of Cambridge

2. http://en.wikipedia.org/wiki/Macintosh_SE My first computer.

3. M y latest computer .

4. Macintosh SE
1987
8 MHz
1 MB
20 MB
7.7 kg
$2,900
Model:
Released:
Processor:
RAM:
Hard Drive:
Weight:
Price:
Samsung Galaxy S4
2013
1.6 GHz x 4
2 GB
16/32 GB
130 g
~$500

5. 1/5: the “basics”
why health via smartphones?
why smartphones for health?

6. why health via smartphones?
health “in the moment” vs. “reconstructed”
ubiquity of technology vs. limited face-to-face

7. Smartphones are incredibly personal devices:
they are not often shared.
Research shows that owners regularly keep their
smartphone within arms length of them.
A. Dey et. al. Getting Closer: An Empirical Investigation of the Proximity of
Users to their Smartphones. In ACM Ubicomp 2011.

8. why smartphones for health?
2/5: interactivity
3/5: sensors
4/5: machine learning

9. 2/5: interactivity *
notifications
check-ins
games
situation-awareness
momentary, context-aware, engaging
* I don't research interaction/design

10. notifications

11. check-ins

12. games

13. situation-aware

14. Emotion Sense Q Sense
mood tracking smoking cessation
Dept. of Psychology Behavioural Sciences

15. check-ins
'volunteering'

16. check-ins
'volunteering'

17. notifications
'prompting'

18. “games”
'guiding'

19. situation-aware
'informing'

20. 2/5: interactive apps
a) active behavioural monitoring
b) momentary assessment
c) information delivery

21. 3/5: smartphone sensors
p assive behavioural monitoring

22. Sensors were originally added to smartphones
for purely functional purposes.
E.g., an accelerometer lets the device know when
to display the screen in landscape mode; the GPS
allows the device to support maps/driving apps.
Only later did researchers uncover that all of
these sensors could be a valuable source of
behavioural data.

23. what is a “sensor?”
Accelerometer
GPS / Wi-Fi
Gyroscope
Bluetooth
Microphone
Environment
Phone / Text Logs
Device Logs
Social Media APIs
App Usage

24. Accelerometer → activity
GPS / Wi-Fi → location, mobility
Gyroscope → device orientation
Bluetooth → co-location
Microphone → audio processing
Environment → light / temperature / pressure
Phone / Text Logs → socialising
Device Logs → network / battery
Social Media APIs → socialising, connectivity
App Usage → context, searching

25. Sensors do not “directly” encode behaviour. For
example, sampling from the accelerometer
provides time series data of changes in
acceleration.
All sensor data needs to be processed in order to
extract/infer behaviours. How, for example, does
the accelerometer indicate physical activity?

26. raw sensor data
features

27. does the accelerometer feature correlate with
reports of current levels of physical activity?
r = 0.369
does the accelerometer feature correlate with
reports of levels of physical activity on that day?
r = 0.172

30. ...batteries are still headaches:
Sensors were originally added to smartphones
for purely functional purposes.
These sensors were not built to efficiently collect
continuous streams of data*.
* This is changing...?
K. Rachuri. Smartphones Based Social Sensing: Adaptive Sampling, Sensing
and Computation Offloading. PhD Thesis, Computer Laboratory. 2012.

31. 4/5: machine learning
u sing data to infer behaviour

32. Machine Learning (vs. Behaviour Theory?)
Behaviours are often too complex and/or
abstract to directly encode them into software.
Machine learning a statistical approach that
centres around using data to learn to identify and
predict behaviours. Often without knowing much
(or anything) about what those behaviours
actually look like.

33. Two broad categories of learning algorithms,
which are often referred to as unsupervised and
supervised learning.

34. Unsupervised learning, or clustering, assumes
you have:
(a) a large dataset of many representations of a
behaviour, and
(b) a way of measuring the extent that two
representations of behaviours are similar.
… without knowing precisely how to code for that
behaviour

35. What are the behaviours that emerge when a city
uses the stations in a bicycle sharing scheme?
Define a way of representing the behaviour:
Station A ... 50% 25% 32%
... 7AM 8AM 11PM

36. What are the behaviours that emerge when a city
uses the stations in a bicycle sharing scheme?
Define a way of comparing behaviour:
Station A ... 50% 25% 32%
...
Station B ... 23% 34% 52%

37. What are the behaviours that emerge when a city
uses the stations in a bicycle sharing scheme?

38. What diurnal patterns of physical activity emerge
from smartphone accelerometers?

39. What diurnal patterns of physical activity emerge
from smartphone accelerometers?
User A ... 1.23 2.33 3.12
... 7AM 8AM 11PM

40. What diurnal patterns of physical activity emerge
from smartphone accelerometers?
User A ... 1.23 2.33 3.12
...
User B ... 2.33 3.43 2.33

41. What diurnal patterns of physical activity emerge
from smartphone accelerometers?
How does this relate to happiness?

42. 4/5: machine learning
u sing data to infer & predict behaviour
* I didn't include supervised learning, which is awesome

43. 5/5: challenges & opportunities

44. 1. Software Engineering / Expectations
2. Marketing
3. Control over target population
4. Understanding sensor data
5. Writing code
6. Finding research value

45. 1. Blurred lines between research and practice
2. High potential for multi-disciplinary impact
3. Cheap to roll-out to huge audiences
4. Accessible to 'everyone'
5. Rising demand for quality healthcare technology
6. Wearables are coming!

46. Opportunities and Challenges of
Using Smartphones for Health
Monitoring and Intervention
@neal_lathia
Computer Laboratory
University of Cambridge

47. “Can I run an ESM study
like Emotion Sense?”