VoIP Service and Marketing using Odoo and Asterisk PBX
Lifelogging, egocentric vision and health: how a small wearable camera can help me improve my health state
1. Petia Radeva
petia.ivanova@ub.edu
University of Barcelona & Computer Vision Center
Lifelogging, egocentric
vision and health: how a
small wearable camera can
help me improve my
health state
22:51 1
2. Social networks & lifelogging
• Facebook.
– Facebook has more than 350 million active users and more than 2.5 billion photos are uploaded
every month.
• Flickr.com
– Flickr has around 44 million users.
• Twitter.com
– More than 3 million tweets are published daily.
• Youtube.
– Youtube is one of the Top-5 most visited websites in the world with more than 5 billion videos
uploaded monthly.
• Instagram
– Instagram exceed 20 billion photos shared
2
“To Log or Not to Log? Privacy Risks and Solutions for Lifelogging and Continuous Activity Sharing Applications”, Blaine Price, ENISA, Centre for Research in
Computing.
22:51
3. The age of the quantified self
22:51 3
Our body is radiating data
Data: loudly, continuously,
honestly and individually
Heart rate, risk for cancer, events that change our mood,
stress, breading, sleep, etc.
Sensors for caregivers, mothers, fermons sensors.
Sensors for behavioural modifications to be more:
mindful, aware, present, … human.
Lauren Constantini: Wearable tech expands
human potential
4. Which wearables do consumers plan
to buy?
• It’s expected to double by 2018, to 81.7 million users.
• Almost 2 in 5 internet users will use wearables by 2019.
22:51 4
The Consumer Technology Association (CTA), formerly the Consumer Electronics Association (CEA), surveyed
1,001 US internet users. Source: eMarketer.
5. Which wearables do consumers plan
to buy?
The Consumer Technology Association (CTA), formerly the Consumer Electronics Association (CEA), surveyed
1,001 US internet users. Source: eMarketer.
“Wearable usage will grow by nearly 60% this year.
Wearables market grows 172% in a year; 78 Million devices shipped (21 million Fitbits)”
IoT Daily, Connected thinking
22:51 5
7. Technology for life-logging is here!
Evolution of life-logging apparatus, including wearable computer, camera, and viewfinder with wireless Internet connection. Early apparatus used
separate transmitting and receiving antennas. Later apparatus evolved toward the appearance of ordinary eyeglasses in the late 1980s and early
1990s .
7
“Quantified Self & life-logging MeetsInternet of Things (IOT)”, Mazzlan Abbas.
22:51
8. Wearable cameras and the life-logging trend
8
Shipments of wearable computing devices
worldwide by category from 2013 to 2015 (in
millions)
Wearable Camera Shipments and Revenue, World
Markets: 2015-2021, Source: Tractica
22:51
9. Visual life-logging
Benefits:
• A digital memory of people you met, conversations you had, places you visited, and events you
participated in.
– This memory would be searchable, retrievable, and shareable.
• A 14/7/365 monitoring of daily activities.
– This data could serve as a warning system and also as a personal base upon which to diagnosis illness and to prescribe medicines.
• A way of organizing, shaping, and “reading” your own life.
– A complete archive of your work and play, and your work habits. Deep comparative analysis of your activities could assist your
productivity, creativity, and consumptivity.
• To the degree this life-log is shared, this archive of information can be leveraged to help others work,
amplify social interactions, and in the biological realm, shared medical logs could rapidly advance
medicine discoveries.
9
The hard part is no longer deciding what to hold on to, but how to efficiently
organize it, sort it, access it, and find patterns and meaning in it.
22:51
10. 10 things to know about lifelogging
1. Social services get logging (Facebook, Twitter, Spotify)
2. Fitness trackers are big market
3. Lifelogging apps for smartphones: Saga, Instant, Narrato, OptimizeMe
4. There is dedicated hardware (wearable cameras)
5. Big tech companies are sniffing around (Sony, Samsung, Apple, etc.)
6. Wearables capture more data (smart watches and augmented glasses)
7. It really isn't a new thing (MyLifeBits since 2000s, Gordon Bell)
8. Lifelogging gets emotional (MIT project)
1. Lifelogging can be art (Alan Kwan's Bad Trip, Stephen Cartwright)
22:51 10
12. Ethical guidelines for wearable
cameras
• Anonimity and confidentiality: Researchers coding image data should:
– not discuss the content with anyone outside of the team,
– not identify anyone they recognize in the images,
– be aware of how sensitive the data are.
• Data encryption: Confıdentiality can be protected by confıguring devices and using
specialist viewing software to make the images accessible only to the research team (lost
devices).
– Devices should be configured so that data can only be retrieved by the research team. It should
be impossible for participants or third parties who find devices to access the images.
• Data storage: Collected images should be stored securely and password-protected,
according to national regulations.
1222:51
13. Understanding user privacy requirements and
risks from emerging technologies
• People are bad at:
1. understanding the future value of revealing private information
today,
2. understanding the risks from technology they have not yet used
or heard of.
13
Wearable cameras can be very useful:
An estimated one million Russian motorists have dashboard video cameras installed in their cars.
Police officers carring video camera units and using Velcro to place these cameras in police wagons,
helmet cams, ear cams, chest cams with audio capability, GPS locators, taser cams
• “Even with only half of the 54 uniformed patrol officers wearing cameras at any given time, a
department in USA had an 88 % decline in the number of complaints filed against officers,
compared with the 12 months before the study”, The New York Times, 4th of July, 2013.
The world’s leading police
body worn video camera
deployed by over 4000
agencies in 16 countries.
22:51
14. Benefits and potential applications
• It will take quite some time for people to feel comfortable with ‘always connected’ devices that
can discreetly take photos or videos. Will the benefits outweigh the negatives?
“Quantified Self & life-logging Meets Internet of Things (IOT)”, Dr. Mazlan Abbas, MIMOS Berhad
1422:51
17. Wealth of life-logging data
• We propose an energy-based approach for
motion-based event segmentation of life-
logging sequences of low temporal resolution
• - The segmentation is reached integrating
different kind of image features and classifiers
into a graph-cut framework to assure
consistent sequence treatment.
Complete dataset of a day captured with SenseCam (more than 4,100 images
17
Choice of devise depends on:
1) where they are set: a hung up camera has
the advantage that is considered more
unobtrusive for the user, or
2) their temporal resolution: a camera with a
low fps will capture less motion information,
but we will need to process less data.
We chose a SenseCam or Narrative - cameras
hung on the neck or pinned on the dress that
capture 2-4 fps.
The wealth or the hell of life-logging data
22:51
18. Extracting semantics from egocentric
images
Computer Vision allows to process and analyze huge
amount of images and extract the semantics from them.
1822:51
What technology to apply?
32. Analysis of human interaction
32
F-formations extraction from egocentric vision
22:51
33. Two main questions?
• What we eat?
– Automatic food recognition vs.
Food diaries
• And how we eat?
– Automatic eating pattern
extraction – when, where, how,
how long, with whom, in which
context?
3322:51
40. What did I do today?
40
Wearable cameras allows to visualize our diary.
Computer Vision allows to process huge amount of data and automatically extracts
the semantics from it: where, what, who, how, etc.
22:51
48. Metabolic diseases and health
4822:51
4.2 million die of chronic diseases
in Europe (diabetes or cancer)
linked to lack of physical activities
and unhealthy diet.
Physical activities can increase
lifespan by 1.5-3.7 years.
Obesity is a chronic disease
associated with huge economic,
social and personal costs.
Risk factors for cancers,
cardiovascular and metabolic
disorders and leading causes of
premature mortality worldwide.
49. Health and medical care
• Today, 88% of healthcare costs are spent
on medical care – access to physicians,
hospitals, procedures, drugs, etc.
• However, medical care only accounts for
approximately 10% of a person’s health.
• Approximately half the decline in U.S.
Deaths from coronary heart disease from
1980 through 2000 may be attributable to
reductions in major risk factors (systolic
blood pressure, smoking, physical
inactivity).
4922:51
50. What are we missing in health
applications?
• Today, automatically measuring physical activity is not a problem.
• But what about food and nutrition?
– State of the art: Nutritional health apps are based on manual food diaries.
22:51 50
Sparkpeople
LoseIt!
MyFitnessPal
Cronometer Fatsecret
51. Towards healthy habits
Towards visualizing summarized lifestyle data to ease the management of the user’s healthy
habits (sedentary lifestyles, nutritional activity, etc.).
Life-logging can help us to extract our nutritional habits: taking photos of our everyday life and being able to analyse
what we eat, starting by the dish recognition, where we eat, with whom we eat, how we eat.
22:51 51
52. Teleassistance and home monitoring
Using wearable cameras, we can have a clear picture of the lifestyle
the person is having during the whole day:
• Is he/she active or passive
– doing domestic duties,
– performing intellectual activities like reading,
– coping with daily functions, etc.
22:51 52
53. Life-logging for cognitive treatment of people with amnesia
53
Program based on life-logging captured by a Wearable Camera recording specific
autobiographical episodes for stimulating posteriorly episodic memory function.
Using wearable cameras and looking at their autobiographic experiences, people with
amnesia improved their memory and cognitive facilities by using episodic images.
Claire, a 49-year-old former nurse, six years ago suffered brain damage due to a rare viral infection called herpes encephalitis. Now an amnesiac
who is unable to recognize faces, Claire lives in a world in which even her lifelong friends appear as strangers.
22:51
54. Life-logging for MCI treatment
54
Goal: using episodic images to develop cognitive exercises and tools for memory
reforcing of MCI and Alzheimer people.
To explore the association between changes in cognitive, functional and
emotional outcomes.
22:51
55. Application of Lifelogging to Migraine
• Objective Biomarkers in Migraine - how to predict a migraine attack
• How does the brain interact with the environment - the cues that ensure
adaptation
55
Lifelogging can assure individualized tools to detect which are the triggers of the migraine for
an individual.
22:51
56. Lifelogging and wearable cameras
• Lifelogging technology is here
• Computer Vision and Machine Learning can
help process big image data
• How to get profit of it to help improve our
health?!
22:51 56
But in fact we are not very interested in every and each of these images but to extract semantic information or actually meanings out of them for later use.
These meanings could be anything, from thinking about how I do eat or workout or even communicate with others. Studying these semantic info is an interesting subject since they can help one to observe his own life style or even helping people with memory impairment to improve their memory by reviewing events happened to them during a day.
To accomplish this, a search engine is required to use existing clues to “recognize” different events and meanings, this procedure is called life-logging
Other methods also use unsupervised pre-training to structure a neural network, making it first learn generally useful feature detectors. Then the network is trained further by supervised back-propagation to classify labeled data. The deep model of Hinton et al. (2006) involves learning the distribution of a high level representation using successive layers of binary or real-valued latent variables. It uses a restricted Boltzmann machine to model each new layer of higher level features. Each new layer guarantees an increase on the lower-bound of the log likelihood of the data, thus improving the model, if trained properly. Once sufficiently many layers have been learned the deep architecture may be used as a generative model by reproducing the data when sampling down the model (an "ancestral pass") from the top level feature activations.[8] Hinton reports that his models are effective feature extractors over high-dimensional, structured data.[9]
Natural Language Processing which is used heavily in language conversion in chat rooms or processing text from where human speeches.
Optical Character Recognition which is scanning of images. It's gaining traction lately to read an image and extract text out of it and correlate to the objects found on image
Speech Recognition applications like Siri or Cortana needs no introduction
Artificial Intelligence induction to different robots for automating at least a minute level of tasks a human can do. We want them to be a little smarter.
Drug discovery though medical imaging-based diagnosis using deep learning. It's kind of in early stages now. Check Butterfly Network for the work they are doing.
CRM needs for companies are growing day by day. There are hundreds of thousands of companies around the globe from small to big companies who wants to know their potential customers. Deep Learning has provided some outstanding results. Check for companies like RelateIQ (product) who has seen astounding success of using Machine Learning in this area.
Medical applications - there are tremendous advances in robotic surgery that relies on extremely sensitive tactile equipment. However, if a doctor can advise a robot to "move a fraction of a millimeter to the left of the clavicle" they could potentially gain more control by directing the robot via full understood voice control.
Automotive - we are already seeing self driving cars; deep learning will possibly integrate into automated driving systems to detect and interpret sights and sounds that might be beyond the capacity of humans.
Military - drones are particularly well suited to deep learning.
Surveillance - here too drones will play a role, but the idea of computers that are able to sense and interpret with a human-like degree of accuracy will change the way in which surveillance is done.