Society is currently going through a phase of having an adversarial relationship with personal data. Our data is gathered by third parties ranging from companies like Facebook and Google to governments and their agencies and although in theory we ourselves own our data, we don’t manage, get value from it, or use it ourselves. The only times we encounter our own data is when we read about abuses of it, or we get confused when we try to understand what GDPR means. One day we will live in a world where we actually own our own data and it will be managed for us, with our interests at heart, by trusted third parties analogous to how banks manage our wealth. Those third parties may increase the value of our data by pooling it, equivalent to banks lending money, and by sharing it with organisations like social media companies, educational institutions, entertainment companies, etc. In such a world we would be delighted rather than afraid, to gather data and to have data gathered about ourselves and used for our benefit. In such a world, what are the data points that can be gathered, what is our digital footprint ? In this talk I will present an overview of what data can, and is gathered by people about themselves. I will cover off-the-self and popular sensors as well as the more unusual and uncommon and as a focus I will give an overview of sleep, how it can be measured and what use that can be. Gathering data about oneself is also known as lifelogging or the quantified self and I will draw inspiration and case studies from the work we have done in the area of lifelogging over the last 15 years. (thanks to Cathal Gurrin for some of the slides).

1. When My Data Actually Becomes My
Data
Alan F. Smeaton
Dublin City University
@asmeaton

2. Who’s Alan Smeaton ?
- DCU Professor for +20 years
- Founding Director of Insight
- Decorated, highly-cited, worked
on 00’s research projects, loads of
PhD graduates
- Background in multimedia
analysis, indexing and search
- Known for his collaborations

3. • We have an adversarial relationship with our personal data
• Usually gathered by third parties and in theory we own our data, but
we don’t manage, get value from, or use it ourselves
• We read about abuses of it
• We can also gather data ourselves, about ourselves, for standalone
niche applications (sleep, steps, energy, screen usage, travel, etc.), or
we can do extreme forms of this called lifelogging
• Lets have a look at that niche application area, and some of the
extreme lifelogging
3

4. Digital Footprints
• Digital Footprints used to be the electronic evidence of a
computer user's activity (online, local, etc.), used for debugging
• Early search engines logged queries/clicks
• Clickthroughs were mined, turned into AdWords, pushed the
boundaries of ML, created Google
• Now a digital footprint is the electronic evidence of your
existence, since everything is now digital and so much is logged
by third parties anyway

5. Digital Footprints
• We’re aware we consciously and unconsciously leave digital
footprints – conscious ones from
• Web searches
• Website visits and cookies
• Internet connections
• Purchases
• Social media check-ins, posts and photos
• These are what we expect Google et al. to know about us …
like all marketing, they use this to segment their market (us) so
we get adverts (like me viewing YouTube videos with fingers
crossed)
• With rich data they segment their market into N=1 by turning
footprints into personality profiles

6. Digital Footprints
• So that’s our digital footprints
• Some we know about, its obvious, and we accept,
we even like it … the Faustian pact
• Some we don’t realise, we’re surprised at, but we’re
OK with
• Some is a bit creepy, maybe crossing a line, intrusive
• Our awareness varies hugely – vast majority don’t
realise, those that do know, don’t actually know it all

7. • Most of the time, 3rd parties gather data about us but
occasionally we gather data about ourselves, we gather it
• That’s called lifelogging
7

8. Depending on what you want
to measure, there is likely to
be a device (or an app)

9. Digitise as much as you
can of life experience…
for many reasons
(memory, health, etc.).
Lifelogging
Sense and analyse
factors of interest
through numbers to gain
knowledge
Using
knowledge for
self-
improvement
through
experimentatio
n
Quantified
Self
Biohackin
g

10. 1928
Bucky Fuller
Archive
1980s
Steve Mann
2004
Williams
(Sensecam)
2010
Quantified-
Self
2006+
Memory
Studies
2004
G. Bell
(MyLifeBits)
1946
Vannevar
Bush

11. Richard Buckminster Fuller
• Interested in “a very accurate record of a human
being" … so he made himself his own case study ..
put everything in and created a very rigorous
record … the Dymaxion Chronofile…
• He documented his life, philosophy and ideas
scrupulously by a diary every 15 minutes, now on
display at the Stanford Library.
11
Richard Buckminster
"Bucky" Fuller was an
American architect,
systems theorist,
author, designer,
inventor and futurist.
Fuller was the second
World President of
Mensa from 1974 to
1983

12. With more than 140,000 papers and 1,700 hours of audio and video, all stretching to more than
1,400 linear feet of material, Fuller’s life might be the most documented life of all time. From 1917
to his death in 1983 he collected all documentation including (mail, newspaper clippings,
drawings, blueprints, models, and even bills.
The Dymaxion Chronofile has been at the Stanford University Libraries Department of Special
Collections since 1999. There you can pick any day of these years of his life and find out exactly
what he was doing nearly to the hour by flipping through a scrapbook.

13. Vannevar Bush (External Memory)
13
Vannevar Bush was an
American engineer,
inventor and science
administrator, who
during World War II
headed the U.S.
wartime military R&D
including initiation of the
Manhattan Project. "As
We May Think" has
turned out to be a
visionary and influential
essay.
https://www.theatlantic.com/magazine/archive/1945/07/as-we-may-think/303881/

14. https://mediartinnovation.com/2014/06/06/vannevar-bush-memex-1945/
Vannevar Bush (External Memory)

15. Steve Mann’s Wearcam
• Steve Mann (University of Toronto) built a wearable
camera called Wearcam (wearcam.org). Steve would
wear the camera and it uploaded images to the WWW
for others to see.
• Mann has been referred to as the "father of wearable
computing", having created the first general-purpose
wearable computer. Mann has also been described as
"the world's first cyborg”.
15
Steven Mann is a
Canadian researcher
and inventor best
known for his work on
augmented reality,
computational
photography,
particularly wearable
computing and high
dynamic range
imaging.

16. 2012
Steve Mann’s Wearcam

17. Gordon Bell
An early employee of Digital Equipment
Corporation (DEC), Bell designed several
PDP machines and later became Vice
President of Engineering, for VAX
computers
He is the experiment subject for the
MyLifeBits project, an experiment in life-
logging and an attempt to fulfill Vannevar
Bush's Memex.
17
Gordon Bell is an
American electrical
engineer, pioneer and
investor. He is the
founder of the
MyLifeBits project, an
experiment in life-
logging based on
Vannevar Bush's
Memex.

18. Lifelog
Life
Experience
The individual will have a lifelog / human
ledger for many aspects of life
experience… activities, experiences,
behaviours, information, biometrics… huge
volumes of data captured passively.
Gordon Bell – the lifelog philosophy

19. Cathal Gurrin
In 2006 he put on a wearable camera, he’s still wears one, every day all
day, 12 years and billions of data points later
His interest is in multimodal multimedia analysis, indexing and retrieval

20. Unstructured Lifelog Data
20

21. Wearable cameras to
capture our activities
21

22. A complete trace of the individual

23. Know their activities and interests

24. Know their habits

25. Know their interests and experiences

26. And know what they consume

27. Devices
Apple Watch & HealthKit
Wahoo Chest Strap
Fitbit Activity Tracker / Scales
Nokia Activity Tracker / Scales / BP
…
Aggregation Software
HealthKit
Microsoft Health
Gyrosco.pe
We’ll come
back to
this

28. Issues with Wearable Camera Images
Huge levels of redundancy
28

29. Issues with Wearable Camera Images
Vary in quality from unusable to photo-album
29

31. Issues with Wearable Camera Images
Don’t typically have
a salient object
Capture the hands
of the wearer
31

32. CATHAL’S LIFELOG
Autographer Panasonic 4K Google Glass
Moves App Withings BASIS Watch Reporter
RescueTime LoggerMan Camlapse
MyTracks OpenPaths
CameraPhoneInstagram
Media Lifelogging
Biometrics/
Activity
Lifelog
Information Access
SMS Backup
Youtube Log
VoiceRecorder
WebServices
Swarm
Location
Digital Paper PDF Anno. Web Pages
Consumption
e-book/mag
Last.fm
NarrativeClip
3
2

33. Volume Growth over a Decade

34. Which allows for the development of various types of interface.
3
4

35. Events with colour coded minutes… showing the dominant colours
3
5

36. Minute-by-
minute
segmentation
and
summarisation of
life activities

37. Segmentation of raw data into
units such as events or moments.
These can be enriched
automatically with metadata,
increasing their value.
Events are analogous to our episodic
memory and can be segmented based
on many forms of data.

38. Quantified Self
Personal Insights
Data-driven Health
Augmented
Wellness
Behaviour Change
Enhanced Security
Population-wide
Analytics
Augmented
Community
Augmenting Human
Memory
Nomenclators
Augmented
Memory
Enhanced Productivity
& Education
Enhanced Interactions
Rich Sharing and
Reminiscing
Augmented
Cognition
Some (Individual) Use Cases for Lifelogging
Health & Wellbeing Memory & Cognition

39. Quantified-Self
Analysis
Self-discovery
Reflect
Contextual
Reminders
Remind
Sousveillance.
Protection of me
and bystanders
Protect
Find an item from
the digital self
Validate a memory
Contextual support
Answer
Reminiscence
Therapy
Social applications
Reminisce
Digital Agents
acting on our
behalf, during life
and after
Represent
The most interesting aspect is the potential for memory support,
where the lifelog works in synergy with your own memory.
Adapted from Abigail J. Sellen and Steve Whittaker. 2010. Beyond total capture: a constructive critique
of lifelogging. Communications of the ACM. 53, 5 (May 2010), 70-77.

40. • Back to OTS or regular quantified self rather than
extreme lifelogging
• Lets have a deeper look at one form of lifelogging
… sleep
40

42. • Sleep is “active”, our brains do not shut down and
are almost as active, cataloging memories
• Sleep 5 stages – wake, relaxed wakefulness, light
sleep, deep sleep and REM sleep
• Starts from N1, goes through N2 to N3 (deep) and
then back up towards REM sleep, there is an
ordering
• If you sleep for 8 hours, you have 5 full cycles
Sleep Explained

43. • Each phase has characteristics
– REM – body paralysed, HR, RR increased,
body temperature drops, vivid dreams, brain
active, towards latter end of the night, memory
consolidation
– N1 – conscious of surroundings, hypnic jerks
– N2 – brief arousals, decreased HR, RR
– N3 or deep sleep – slowest HR, RR, difficult to
wake and then groggy
Sleep Explained

44. • Insufficient sleep makes you more
stupid, fatter, unhappier, poorer, sicker,
worse at sex, more grumpy, more likely
to get cancer, Alzheimer’s and to die in
a car crash !
• Recent years have focused on this,
we’re more aware because we
ourselves can now measure it
• [ Orthosomnia — a preoccupation with
perfecting one’s sleep data ]
Why is sleep important ?

45. • Sleep labs record EEG,
body and eye movement,
HR, HRV, RR, Oxygen
saturation, etc.
• They pool all these into a
polysomnograph for an
holistic overview, which
looks like …
Measuring Sleep (Properly)

47. 1. Phone apps
2. Wrist-worn accelerometer devices
3. Movement radar or sonar
4. The Ōuru ring
Each of these uses a subset of sensors
Sensing Sleep – Our Options

48. Presented as a Hypnogram

49. • How ? They record ..
– Movement (in the bed, under pillow)
– Microphone (listening to your breathing)
– Sonar – inaudible frequency emitted and listened to, just like bats !
• There are many available, some freebies, some
paid … SleepScore, Sleep Cycle, Pillow, etc.
1. Smartphone Apps

50. • FitBit, Jawbone, Withings,
LARK, etc.
• These “just” do movement
but directly, so more
accurately than phone apps
2. Wrist-worn Accelerometers

51. • BiancaMed -> ResMed -> S+ was first to market
• Very low levels of transmitted radio-frequency
power acting as sonar
• Contactless, measures
motion, plus room
temperature, brightness
and noise level (from
phone, yes, phone is
listening as you sleep)
3. Sonar / Radar

52. 4. The Ōura Ring
• Includes accelerometer,
gyroscope, temperature, and
heart rate
• HR sampled using IR spectrum
at 250 Hz so able to do much
more than wrist-worn
• Contactless battery charge lasts
7 days, data capacity 6 weeks
• Low power Bluetooth download
to phone

53. • Consumer grade wearables are not sleep labs,
they use a proxy for an orchestra of sensors !
• Each individual sensor will have errors
(movement, sweat, etc.)
• The algorithms to compute “sleep efficiency” are
opaque and proprietary
• I compared S+ with Ōura for me over 8 weeks and
…
Accuracy of sleep tracking ?

54. • 56 days, but S+ not continuous (travel, S+ is not
travel-friendly) – I missed 1, 1 and 7 days
• Correlation is (only) 0.147 (rises to 0.16 when
blanks eliminated)
• And because ranges may not be normalized, the
visualization reveals …
S+ and Ōura

55. S+ and Ōura - 56 days continuous – 0.147

56. So What ?
• Its all a bit … unsatisfactory
• I’m feeling a bit … let down … it shouldn’t be like this,
there’s something not right
• I’m actually feeling … trapped .. its my data, about
me, but I can’t fix this, for me, I can’t leverage benefit,
• Not everyone is like me though, and would want to do
this, but maybe you’d want somebody to do this for
you, like a wealth manager or stock broker
• There are some benefits at population level
56

57. Almost all vendors who allow us gather individual
data, get value from pooling anonymised data
• Fitbit – 150B hours of HR data, statistics from
millions of sleep nights
• 23a DNA testing – diseases, long-lost relatives
• Strava – global heatmap of runs/cycles
• Jaw – pinpointing earthquake epicentres !
Population-level Analytics from Pooled
Individual Data

58. Resting Heart Rate is an excellent indicator of overall health.
https://finance.yahoo.com/news/exclusive-fitbits-150-billion-hours-heart-data-reveals-secrets-human-health-133124215.html

59. Strava Global Heatmap

60. 2014 – Northern Calif. 6.0M Earthquake
% people who woke up correlates with
distance from epicentre

63. Back to my problem of …
• Unsatisfactory … let down … trapped
63

64. • Open up about their algorithms and allow a
“normalisation” of sleep metrics for interoperability
and device/vendor independence
• Include “errorbars” in their outputs
• Allow exporting of our data, our original raw data,
with those error margins
Sleep Sensing Vendors ought to …

65. • Imagine a world where our data is managed for us by trusted
third parties analogous to how banks manage wealth
• They may pool it, like banks lend money
• They may share mine with organisations like social media
companies, educational institutions, entertainment companies,
etc. but only if I say so, and I might get paid, or I might get better
services
• I would be delighted rather than afraid, to gather data and have
data gathered about me for my benefit
• Most people would not micromanage data, most people don’t
micromanage wealth
• We have seen the data points that can be gathered
Utopia ? Naivete ?

66. • What’s the appetite for this ?
• Is there even a demand ?
• Are there green shoots against surveillance
capitalism ?

67. • DuckDuckGo, StartPage, SearX.me,
DisconnectSearch, MetaGer, Quant, all trace-free
search engines
Trace-free services

68. Devices
Apple Watch & HealthKit
Wahoo Chest Strap
Fitbit Activity Tracker / Scales
Nokia Activity Tracker / Scales / BP
…
Aggregation Software
HealthKit
Microsoft Health
Gyrosco.pe
Remember
this ?

69. 6
9
• Consolidates health data from iPhone,
Apple Watch, and third-party apps
• Activity, Sleep, Mindfulness, and
Nutrition
• “You are in charge of your data”
• “The Health app lets you keep all your
health and fitness information under
your control and in one place on your
device. You decide which information is
placed in Health and which apps can
access your data through the
Health app”

70. dacadoo

71. dacadoo

72. dacadoo

73. dacadoo

74. dacadoo

75. dacadoo
• Revenue stream is
corporate
• Employer signs up
as a service to
employees,
employees use it,
employer gets
healthier (more
productive) staff

76. • I don’t see a spanner breaking up Surveillance
Capitalism, too traumatic
• I see a slow creep towards
– Surveillance Socialism (don’t like that)
– Surveillance Democracy (don’t like that either)
– Sousveillance
• Not happening overnight, generational