SlideShare a Scribd company logo
1 of 58
Download to read offline
Petia Radeva
petia.ivanova@ub.edu
University of Barcelona & Computer Vision Center
Lifelogging, egocentric
vision and health: how a
small wearable camera can
help me improve my
health state
22:51 1
Social networks & lifelogging
• Facebook.
– Facebook has more than 350 million active users and more than 2.5 billion photos are uploaded
every month.
• Flickr.com
– Flickr has around 44 million users.
• Twitter.com
– More than 3 million tweets are published daily.
• Youtube.
– Youtube is one of the Top-5 most visited websites in the world with more than 5 billion videos
uploaded monthly.
• Instagram
– Instagram exceed 20 billion photos shared
2
“To Log or Not to Log? Privacy Risks and Solutions for Lifelogging and Continuous Activity Sharing Applications”, Blaine Price, ENISA, Centre for Research in
Computing.
22:51
The age of the quantified self
22:51 3
Our body is radiating data
Data: loudly, continuously,
honestly and individually
Heart rate, risk for cancer, events that change our mood,
stress, breading, sleep, etc.
Sensors for caregivers, mothers, fermons sensors.
Sensors for behavioural modifications to be more:
mindful, aware, present, … human.
Lauren Constantini: Wearable tech expands
human potential
Which wearables do consumers plan
to buy?
• It’s expected to double by 2018, to 81.7 million users.
• Almost 2 in 5 internet users will use wearables by 2019.
22:51 4
The Consumer Technology Association (CTA), formerly the Consumer Electronics Association (CEA), surveyed
1,001 US internet users. Source: eMarketer.
Which wearables do consumers plan
to buy?
The Consumer Technology Association (CTA), formerly the Consumer Electronics Association (CEA), surveyed
1,001 US internet users. Source: eMarketer.
“Wearable usage will grow by nearly 60% this year.
Wearables market grows 172% in a year; 78 Million devices shipped (21 million Fitbits)”
IoT Daily, Connected thinking
22:51 5
Lifelogging
22:51 6
Technology for life-logging is here!
Evolution of life-logging apparatus, including wearable computer, camera, and viewfinder with wireless Internet connection. Early apparatus used
separate transmitting and receiving antennas. Later apparatus evolved toward the appearance of ordinary eyeglasses in the late 1980s and early
1990s .
7
“Quantified Self & life-logging MeetsInternet of Things (IOT)”, Mazzlan Abbas.
22:51
Wearable cameras and the life-logging trend
8
Shipments of wearable computing devices
worldwide by category from 2013 to 2015 (in
millions)
Wearable Camera Shipments and Revenue, World
Markets: 2015-2021, Source: Tractica
22:51
Visual life-logging
Benefits:
• A digital memory of people you met, conversations you had, places you visited, and events you
participated in.
– This memory would be searchable, retrievable, and shareable.
• A 14/7/365 monitoring of daily activities.
– This data could serve as a warning system and also as a personal base upon which to diagnosis illness and to prescribe medicines.
• A way of organizing, shaping, and “reading” your own life.
– A complete archive of your work and play, and your work habits. Deep comparative analysis of your activities could assist your
productivity, creativity, and consumptivity.
• To the degree this life-log is shared, this archive of information can be leveraged to help others work,
amplify social interactions, and in the biological realm, shared medical logs could rapidly advance
medicine discoveries.
9
The hard part is no longer deciding what to hold on to, but how to efficiently
organize it, sort it, access it, and find patterns and meaning in it.
22:51
10 things to know about lifelogging
1. Social services get logging (Facebook, Twitter, Spotify)
2. Fitness trackers are big market
3. Lifelogging apps for smartphones: Saga, Instant, Narrato, OptimizeMe
4. There is dedicated hardware (wearable cameras)
5. Big tech companies are sniffing around (Sony, Samsung, Apple, etc.)
6. Wearables capture more data (smart watches and augmented glasses)
7. It really isn't a new thing (MyLifeBits since 2000s, Gordon Bell)
8. Lifelogging gets emotional (MIT project)
1. Lifelogging can be art (Alan Kwan's Bad Trip, Stephen Cartwright)
22:51 10
But!
22:51 11
Ethical guidelines for wearable
cameras
• Anonimity and confidentiality: Researchers coding image data should:
– not discuss the content with anyone outside of the team,
– not identify anyone they recognize in the images,
– be aware of how sensitive the data are.
• Data encryption: Confıdentiality can be protected by confıguring devices and using
specialist viewing software to make the images accessible only to the research team (lost
devices).
– Devices should be configured so that data can only be retrieved by the research team. It should
be impossible for participants or third parties who find devices to access the images.
• Data storage: Collected images should be stored securely and password-protected,
according to national regulations.
1222:51
Understanding user privacy requirements and
risks from emerging technologies
• People are bad at:
1. understanding the future value of revealing private information
today,
2. understanding the risks from technology they have not yet used
or heard of.
13
Wearable cameras can be very useful:
An estimated one million Russian motorists have dashboard video cameras installed in their cars.
Police officers carring video camera units and using Velcro to place these cameras in police wagons,
helmet cams, ear cams, chest cams with audio capability, GPS locators, taser cams
• “Even with only half of the 54 uniformed patrol officers wearing cameras at any given time, a
department in USA had an 88 % decline in the number of complaints filed against officers,
compared with the 12 months before the study”, The New York Times, 4th of July, 2013.
The world’s leading police
body worn video camera
deployed by over 4000
agencies in 16 countries.
22:51
Benefits and potential applications
• It will take quite some time for people to feel comfortable with ‘always connected’ devices that
can discreetly take photos or videos. Will the benefits outweigh the negatives?
“Quantified Self & life-logging Meets Internet of Things (IOT)”, Dr. Mazlan Abbas, MIMOS Berhad
1422:51
How else can be LL useful?
1522:51
Life-logging data
16
• What we have:
22:51
Wealth of life-logging data
• We propose an energy-based approach for
motion-based event segmentation of life-
logging sequences of low temporal resolution
• - The segmentation is reached integrating
different kind of image features and classifiers
into a graph-cut framework to assure
consistent sequence treatment.
Complete dataset of a day captured with SenseCam (more than 4,100 images
17
Choice of devise depends on:
1) where they are set: a hung up camera has
the advantage that is considered more
unobtrusive for the user, or
2) their temporal resolution: a camera with a
low fps will capture less motion information,
but we will need to process less data.
We chose a SenseCam or Narrative - cameras
hung on the neck or pinned on the dress that
capture 2-4 fps.
The wealth or the hell of life-logging data
22:51
Extracting semantics from egocentric
images
Computer Vision allows to process and analyze huge
amount of images and extract the semantics from them.
1822:51
What technology to apply?
22:51 19
Deep leearning everywhere
2022:51
Deep learning applications
2122:51
From NN to CNN
2222:51
Imagenet
2322:51
ConvNets are everywhere
24From: Fei-Fei Li & Andrej Karpathy & Justin Johnson22:51
ConvNets are everywhere
25From: Fei-Fei Li & Andrej Karpathy & Justin Johnson22:51
ConvNets are everywhere
26From: Fei-Fei Li & Andrej Karpathy & Justin Johnson22:51
ConvNets are everywhere
27From: Fei-Fei Li & Andrej Karpathy & Justin Johnson22:51
ConvNets are everywhere
28From: Fei-Fei Li & Andrej Karpathy & Justin Johnson22:51
Egocentric vision
• How can deep learning help process
egocentric images?
22:51 29
Informativeness
CNN
Day Lifelog
Informative
Images
Egocentric informativeness through deep
learning
22:51 30
Visual summary through keyframe extraction (joint work UPC &
UB)
31
Results
22:51
Analysis of human interaction
32
F-formations extraction from egocentric vision
22:51
Two main questions?
• What we eat?
– Automatic food recognition vs.
Food diaries
• And how we eat?
– Automatic eating pattern
extraction – when, where, how,
how long, with whom, in which
context?
3322:51
Automatic Food Analysis
• Food detection
• Food recognition
• Food environment recognition
• Eating pattern extraction
3422:51
Food datasets
35
Food256 - 100images/class
Classes: 256
Food101 – 101.000 images
Classes: 101
Food101+FoodCAT: 146.392 (101.000+45.392)
Classes: 131
EgocentricFood: 5038 images
Classes: 9
22:51
Food-related environment
recognition
Food
Non Food
...
w1
w2
wn
G
oogleNet
Softm
ax
G
AP
inception4eoutput
Deep
Convolution
X
FAM
Bounding
Box
G
eneration
36
Our localization method based on Global Average Pooling (GAP) that produces a Food Activation Map
(FAM).
Examples of localization and recognition on UECFood256 (top) and EgocentricFood (bottom). Ground
truth is shown in green and our method in blue.
Marc Bolaños, Petia Radeva: Simultaneous Food Localization and Recognition. submitted to ICPR’16, arXiv.org>cs>arXiv:1604.07953, 2016.
22:51
Food environment classification
37
Bakery
Banquet hall
Bar
Butcher shop
Cafetería
Ice cream parlor
Kitchen
Kitchenette
Market
Pantry
Picnic Area
Restaurant
Restaurant Kitchen
Restaurant Patio
Supermarket
Candy store
Coffee shop
Dinette
Dining room
Food court
Galley
Classification results:
0.92 - Food-related vs. Non-food-related
0.68 - 22 classes of Food-related categories22:51
Image Input
Foodness Map
Extraction
Food Detection CNN
Food Recognition CNN
Food Type
Recognition
Apple
Strawberry
Food recognition
Results: TOP-1 74.7%
TOP-5 91.6%
SoA (Bossard,2014): TOP-1 56,4%
22:51 38
Demo
3922:51
What did I do today?
40
Wearable cameras allows to visualize our diary.
Computer Vision allows to process huge amount of data and automatically extracts
the semantics from it: where, what, who, how, etc.
22:51
Will life-logging and internet of things help know us better?
4122:51
How much time I spend by day on different
activities?
4222:51
My working hours
4322:51
With whom I was interacting with?
4422:51
How stressful am I in different places?
4522:51
“Six Months of My Life”
46by David El Achkar22:51
4722:51
Metabolic diseases and health
4822:51
 4.2 million die of chronic diseases
in Europe (diabetes or cancer)
linked to lack of physical activities
and unhealthy diet.
 Physical activities can increase
lifespan by 1.5-3.7 years.
 Obesity is a chronic disease
associated with huge economic,
social and personal costs.
 Risk factors for cancers,
cardiovascular and metabolic
disorders and leading causes of
premature mortality worldwide.
Health and medical care
• Today, 88% of healthcare costs are spent
on medical care – access to physicians,
hospitals, procedures, drugs, etc.
• However, medical care only accounts for
approximately 10% of a person’s health.
• Approximately half the decline in U.S.
Deaths from coronary heart disease from
1980 through 2000 may be attributable to
reductions in major risk factors (systolic
blood pressure, smoking, physical
inactivity).
4922:51
What are we missing in health
applications?
• Today, automatically measuring physical activity is not a problem.
• But what about food and nutrition?
– State of the art: Nutritional health apps are based on manual food diaries.
22:51 50
Sparkpeople
LoseIt!
MyFitnessPal
Cronometer Fatsecret
Towards healthy habits
Towards visualizing summarized lifestyle data to ease the management of the user’s healthy
habits (sedentary lifestyles, nutritional activity, etc.).
Life-logging can help us to extract our nutritional habits: taking photos of our everyday life and being able to analyse
what we eat, starting by the dish recognition, where we eat, with whom we eat, how we eat.
22:51 51
Teleassistance and home monitoring
Using wearable cameras, we can have a clear picture of the lifestyle
the person is having during the whole day:
• Is he/she active or passive
– doing domestic duties,
– performing intellectual activities like reading,
– coping with daily functions, etc.
22:51 52
Life-logging for cognitive treatment of people with amnesia
53
Program based on life-logging captured by a Wearable Camera recording specific
autobiographical episodes for stimulating posteriorly episodic memory function.
Using wearable cameras and looking at their autobiographic experiences, people with
amnesia improved their memory and cognitive facilities by using episodic images.
Claire, a 49-year-old former nurse, six years ago suffered brain damage due to a rare viral infection called herpes encephalitis. Now an amnesiac
who is unable to recognize faces, Claire lives in a world in which even her lifelong friends appear as strangers.
22:51
Life-logging for MCI treatment
54
Goal: using episodic images to develop cognitive exercises and tools for memory
reforcing of MCI and Alzheimer people.
To explore the association between changes in cognitive, functional and
emotional outcomes.
22:51
Application of Lifelogging to Migraine
• Objective Biomarkers in Migraine - how to predict a migraine attack
• How does the brain interact with the environment - the cues that ensure
adaptation
55
Lifelogging can assure individualized tools to detect which are the triggers of the migraine for
an individual.
22:51
Lifelogging and wearable cameras
• Lifelogging technology is here
• Computer Vision and Machine Learning can
help process big image data
• How to get profit of it to help improve our
health?!
22:51 56
Thank you!
5722:51
Deep learning applications
5822:51

More Related Content

Similar to Lifelogging, egocentric vision and health: how a small wearable camera can help me improve my health state

Self-Knowledge through Self-Tracking Devices
Self-Knowledge through Self-Tracking DevicesSelf-Knowledge through Self-Tracking Devices
Self-Knowledge through Self-Tracking DevicesBurcu Avci
 
Smart Data for you and me: Personalized and Actionable Physical Cyber Social ...
Smart Data for you and me: Personalized and Actionable Physical Cyber Social ...Smart Data for you and me: Personalized and Actionable Physical Cyber Social ...
Smart Data for you and me: Personalized and Actionable Physical Cyber Social ...Amit Sheth
 
Raai 2019 clinical unmet needs and its solutions of deep learning in medicine3
Raai 2019 clinical unmet needs and its solutions of deep learning in medicine3Raai 2019 clinical unmet needs and its solutions of deep learning in medicine3
Raai 2019 clinical unmet needs and its solutions of deep learning in medicine3Namkug Kim
 
The Future of Neuroimaging: A 3D Exploration of TBI
The Future of Neuroimaging: A 3D Exploration of TBIThe Future of Neuroimaging: A 3D Exploration of TBI
The Future of Neuroimaging: A 3D Exploration of TBIHunter Whitney
 
Data is the new oil: Big data, data mining and bio - inspiring techniques
Data is the new oil: Big data, data mining and bio - inspiring techniquesData is the new oil: Big data, data mining and bio - inspiring techniques
Data is the new oil: Big data, data mining and bio - inspiring techniquesAboul Ella Hassanien
 
Data are the new oil: Big data, data mining and bio - inspiring techniques
Data are the new oil: Big data, data mining and bio - inspiring techniquesData are the new oil: Big data, data mining and bio - inspiring techniques
Data are the new oil: Big data, data mining and bio - inspiring techniquesAboul Ella Hassanien
 
Renish Dadhaniya - GlobeSync Technologies | Work at a glance
Renish Dadhaniya - GlobeSync Technologies | Work at a glanceRenish Dadhaniya - GlobeSync Technologies | Work at a glance
Renish Dadhaniya - GlobeSync Technologies | Work at a glanceGlobeSync Technologies
 
Mobile computing and your future old ppt
Mobile computing and your future old pptMobile computing and your future old ppt
Mobile computing and your future old pptMary Markland
 
Human Activity Recognition
Human Activity RecognitionHuman Activity Recognition
Human Activity RecognitionIRJET Journal
 
Seminar on Artificial Intelligence in Healthcare.pptx
Seminar on Artificial Intelligence in Healthcare.pptxSeminar on Artificial Intelligence in Healthcare.pptx
Seminar on Artificial Intelligence in Healthcare.pptxIsaacLalrawngbawla1
 
Event processing within the human body - Tutorial
Event processing within the human body - Tutorial Event processing within the human body - Tutorial
Event processing within the human body - Tutorial Opher Etzion
 
YOLOv4: A Face Mask Detection System
YOLOv4: A Face Mask Detection SystemYOLOv4: A Face Mask Detection System
YOLOv4: A Face Mask Detection SystemIRJET Journal
 
On how new IT frontiers will impact everyday life: the NECSTLab example
On how new IT frontiers will impact everyday life: the NECSTLab exampleOn how new IT frontiers will impact everyday life: the NECSTLab example
On how new IT frontiers will impact everyday life: the NECSTLab exampleNECST Lab @ Politecnico di Milano
 
The Consequences of Living and Breathing with Hyperconnectedness
The Consequences of Living and Breathing with HyperconnectednessThe Consequences of Living and Breathing with Hyperconnectedness
The Consequences of Living and Breathing with HyperconnectednessFoCAS Initiative
 
Artificial intelligence in healthcare.pptx
Artificial intelligence in healthcare.pptxArtificial intelligence in healthcare.pptx
Artificial intelligence in healthcare.pptxRajesh De
 
From computational Thinking to computational Action - Dr. Hal Abelson, MIT Ap...
From computational Thinking to computational Action - Dr. Hal Abelson, MIT Ap...From computational Thinking to computational Action - Dr. Hal Abelson, MIT Ap...
From computational Thinking to computational Action - Dr. Hal Abelson, MIT Ap...CAVEDU Education
 
Brain Computer Interface and Artificial Brain: Interfacing Microelectronics a...
Brain Computer Interface and Artificial Brain: Interfacing Microelectronics a...Brain Computer Interface and Artificial Brain: Interfacing Microelectronics a...
Brain Computer Interface and Artificial Brain: Interfacing Microelectronics a...Lk Rigor
 

Similar to Lifelogging, egocentric vision and health: how a small wearable camera can help me improve my health state (20)

Multimedia Privacy
Multimedia PrivacyMultimedia Privacy
Multimedia Privacy
 
What is Media in MIT Media Lab, Why 'Camera Culture'
What is Media in MIT Media Lab, Why 'Camera Culture'What is Media in MIT Media Lab, Why 'Camera Culture'
What is Media in MIT Media Lab, Why 'Camera Culture'
 
The_Docs
The_DocsThe_Docs
The_Docs
 
Self-Knowledge through Self-Tracking Devices
Self-Knowledge through Self-Tracking DevicesSelf-Knowledge through Self-Tracking Devices
Self-Knowledge through Self-Tracking Devices
 
Smart Data for you and me: Personalized and Actionable Physical Cyber Social ...
Smart Data for you and me: Personalized and Actionable Physical Cyber Social ...Smart Data for you and me: Personalized and Actionable Physical Cyber Social ...
Smart Data for you and me: Personalized and Actionable Physical Cyber Social ...
 
Raai 2019 clinical unmet needs and its solutions of deep learning in medicine3
Raai 2019 clinical unmet needs and its solutions of deep learning in medicine3Raai 2019 clinical unmet needs and its solutions of deep learning in medicine3
Raai 2019 clinical unmet needs and its solutions of deep learning in medicine3
 
The Future of Neuroimaging: A 3D Exploration of TBI
The Future of Neuroimaging: A 3D Exploration of TBIThe Future of Neuroimaging: A 3D Exploration of TBI
The Future of Neuroimaging: A 3D Exploration of TBI
 
Data is the new oil: Big data, data mining and bio - inspiring techniques
Data is the new oil: Big data, data mining and bio - inspiring techniquesData is the new oil: Big data, data mining and bio - inspiring techniques
Data is the new oil: Big data, data mining and bio - inspiring techniques
 
Data are the new oil: Big data, data mining and bio - inspiring techniques
Data are the new oil: Big data, data mining and bio - inspiring techniquesData are the new oil: Big data, data mining and bio - inspiring techniques
Data are the new oil: Big data, data mining and bio - inspiring techniques
 
Renish Dadhaniya - GlobeSync Technologies | Work at a glance
Renish Dadhaniya - GlobeSync Technologies | Work at a glanceRenish Dadhaniya - GlobeSync Technologies | Work at a glance
Renish Dadhaniya - GlobeSync Technologies | Work at a glance
 
Mobile computing and your future old ppt
Mobile computing and your future old pptMobile computing and your future old ppt
Mobile computing and your future old ppt
 
Human Activity Recognition
Human Activity RecognitionHuman Activity Recognition
Human Activity Recognition
 
Seminar on Artificial Intelligence in Healthcare.pptx
Seminar on Artificial Intelligence in Healthcare.pptxSeminar on Artificial Intelligence in Healthcare.pptx
Seminar on Artificial Intelligence in Healthcare.pptx
 
Event processing within the human body - Tutorial
Event processing within the human body - Tutorial Event processing within the human body - Tutorial
Event processing within the human body - Tutorial
 
YOLOv4: A Face Mask Detection System
YOLOv4: A Face Mask Detection SystemYOLOv4: A Face Mask Detection System
YOLOv4: A Face Mask Detection System
 
On how new IT frontiers will impact everyday life: the NECSTLab example
On how new IT frontiers will impact everyday life: the NECSTLab exampleOn how new IT frontiers will impact everyday life: the NECSTLab example
On how new IT frontiers will impact everyday life: the NECSTLab example
 
The Consequences of Living and Breathing with Hyperconnectedness
The Consequences of Living and Breathing with HyperconnectednessThe Consequences of Living and Breathing with Hyperconnectedness
The Consequences of Living and Breathing with Hyperconnectedness
 
Artificial intelligence in healthcare.pptx
Artificial intelligence in healthcare.pptxArtificial intelligence in healthcare.pptx
Artificial intelligence in healthcare.pptx
 
From computational Thinking to computational Action - Dr. Hal Abelson, MIT Ap...
From computational Thinking to computational Action - Dr. Hal Abelson, MIT Ap...From computational Thinking to computational Action - Dr. Hal Abelson, MIT Ap...
From computational Thinking to computational Action - Dr. Hal Abelson, MIT Ap...
 
Brain Computer Interface and Artificial Brain: Interfacing Microelectronics a...
Brain Computer Interface and Artificial Brain: Interfacing Microelectronics a...Brain Computer Interface and Artificial Brain: Interfacing Microelectronics a...
Brain Computer Interface and Artificial Brain: Interfacing Microelectronics a...
 

Recently uploaded

UiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPathCommunity
 
How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?IES VE
 
Introduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxIntroduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxMatsuo Lab
 
Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Adtran
 
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAAnypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAshyamraj55
 
Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024D Cloud Solutions
 
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...DianaGray10
 
NIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopNIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopBachir Benyammi
 
COMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a WebsiteCOMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a Websitedgelyza
 
UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6DianaGray10
 
Bird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemBird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemAsko Soukka
 
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDEADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDELiveplex
 
Building AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptxBuilding AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptxUdaiappa Ramachandran
 
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Commit University
 
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsIgniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsSafe Software
 
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Will Schroeder
 
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostKubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostMatt Ray
 
Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1DianaGray10
 
VoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBXVoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBXTarek Kalaji
 

Recently uploaded (20)

UiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation Developers
 
How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?
 
Introduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxIntroduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptx
 
Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™
 
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAAnypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
 
Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024
 
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
 
NIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopNIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 Workshop
 
COMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a WebsiteCOMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a Website
 
UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6
 
20150722 - AGV
20150722 - AGV20150722 - AGV
20150722 - AGV
 
Bird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemBird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystem
 
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDEADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
 
Building AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptxBuilding AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptx
 
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)
 
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsIgniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
 
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
 
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostKubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
 
Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1
 
VoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBXVoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBX
 

Lifelogging, egocentric vision and health: how a small wearable camera can help me improve my health state

  • 1. Petia Radeva petia.ivanova@ub.edu University of Barcelona & Computer Vision Center Lifelogging, egocentric vision and health: how a small wearable camera can help me improve my health state 22:51 1
  • 2. Social networks & lifelogging • Facebook. – Facebook has more than 350 million active users and more than 2.5 billion photos are uploaded every month. • Flickr.com – Flickr has around 44 million users. • Twitter.com – More than 3 million tweets are published daily. • Youtube. – Youtube is one of the Top-5 most visited websites in the world with more than 5 billion videos uploaded monthly. • Instagram – Instagram exceed 20 billion photos shared 2 “To Log or Not to Log? Privacy Risks and Solutions for Lifelogging and Continuous Activity Sharing Applications”, Blaine Price, ENISA, Centre for Research in Computing. 22:51
  • 3. The age of the quantified self 22:51 3 Our body is radiating data Data: loudly, continuously, honestly and individually Heart rate, risk for cancer, events that change our mood, stress, breading, sleep, etc. Sensors for caregivers, mothers, fermons sensors. Sensors for behavioural modifications to be more: mindful, aware, present, … human. Lauren Constantini: Wearable tech expands human potential
  • 4. Which wearables do consumers plan to buy? • It’s expected to double by 2018, to 81.7 million users. • Almost 2 in 5 internet users will use wearables by 2019. 22:51 4 The Consumer Technology Association (CTA), formerly the Consumer Electronics Association (CEA), surveyed 1,001 US internet users. Source: eMarketer.
  • 5. Which wearables do consumers plan to buy? The Consumer Technology Association (CTA), formerly the Consumer Electronics Association (CEA), surveyed 1,001 US internet users. Source: eMarketer. “Wearable usage will grow by nearly 60% this year. Wearables market grows 172% in a year; 78 Million devices shipped (21 million Fitbits)” IoT Daily, Connected thinking 22:51 5
  • 7. Technology for life-logging is here! Evolution of life-logging apparatus, including wearable computer, camera, and viewfinder with wireless Internet connection. Early apparatus used separate transmitting and receiving antennas. Later apparatus evolved toward the appearance of ordinary eyeglasses in the late 1980s and early 1990s . 7 “Quantified Self & life-logging MeetsInternet of Things (IOT)”, Mazzlan Abbas. 22:51
  • 8. Wearable cameras and the life-logging trend 8 Shipments of wearable computing devices worldwide by category from 2013 to 2015 (in millions) Wearable Camera Shipments and Revenue, World Markets: 2015-2021, Source: Tractica 22:51
  • 9. Visual life-logging Benefits: • A digital memory of people you met, conversations you had, places you visited, and events you participated in. – This memory would be searchable, retrievable, and shareable. • A 14/7/365 monitoring of daily activities. – This data could serve as a warning system and also as a personal base upon which to diagnosis illness and to prescribe medicines. • A way of organizing, shaping, and “reading” your own life. – A complete archive of your work and play, and your work habits. Deep comparative analysis of your activities could assist your productivity, creativity, and consumptivity. • To the degree this life-log is shared, this archive of information can be leveraged to help others work, amplify social interactions, and in the biological realm, shared medical logs could rapidly advance medicine discoveries. 9 The hard part is no longer deciding what to hold on to, but how to efficiently organize it, sort it, access it, and find patterns and meaning in it. 22:51
  • 10. 10 things to know about lifelogging 1. Social services get logging (Facebook, Twitter, Spotify) 2. Fitness trackers are big market 3. Lifelogging apps for smartphones: Saga, Instant, Narrato, OptimizeMe 4. There is dedicated hardware (wearable cameras) 5. Big tech companies are sniffing around (Sony, Samsung, Apple, etc.) 6. Wearables capture more data (smart watches and augmented glasses) 7. It really isn't a new thing (MyLifeBits since 2000s, Gordon Bell) 8. Lifelogging gets emotional (MIT project) 1. Lifelogging can be art (Alan Kwan's Bad Trip, Stephen Cartwright) 22:51 10
  • 12. Ethical guidelines for wearable cameras • Anonimity and confidentiality: Researchers coding image data should: – not discuss the content with anyone outside of the team, – not identify anyone they recognize in the images, – be aware of how sensitive the data are. • Data encryption: Confıdentiality can be protected by confıguring devices and using specialist viewing software to make the images accessible only to the research team (lost devices). – Devices should be configured so that data can only be retrieved by the research team. It should be impossible for participants or third parties who find devices to access the images. • Data storage: Collected images should be stored securely and password-protected, according to national regulations. 1222:51
  • 13. Understanding user privacy requirements and risks from emerging technologies • People are bad at: 1. understanding the future value of revealing private information today, 2. understanding the risks from technology they have not yet used or heard of. 13 Wearable cameras can be very useful: An estimated one million Russian motorists have dashboard video cameras installed in their cars. Police officers carring video camera units and using Velcro to place these cameras in police wagons, helmet cams, ear cams, chest cams with audio capability, GPS locators, taser cams • “Even with only half of the 54 uniformed patrol officers wearing cameras at any given time, a department in USA had an 88 % decline in the number of complaints filed against officers, compared with the 12 months before the study”, The New York Times, 4th of July, 2013. The world’s leading police body worn video camera deployed by over 4000 agencies in 16 countries. 22:51
  • 14. Benefits and potential applications • It will take quite some time for people to feel comfortable with ‘always connected’ devices that can discreetly take photos or videos. Will the benefits outweigh the negatives? “Quantified Self & life-logging Meets Internet of Things (IOT)”, Dr. Mazlan Abbas, MIMOS Berhad 1422:51
  • 15. How else can be LL useful? 1522:51
  • 17. Wealth of life-logging data • We propose an energy-based approach for motion-based event segmentation of life- logging sequences of low temporal resolution • - The segmentation is reached integrating different kind of image features and classifiers into a graph-cut framework to assure consistent sequence treatment. Complete dataset of a day captured with SenseCam (more than 4,100 images 17 Choice of devise depends on: 1) where they are set: a hung up camera has the advantage that is considered more unobtrusive for the user, or 2) their temporal resolution: a camera with a low fps will capture less motion information, but we will need to process less data. We chose a SenseCam or Narrative - cameras hung on the neck or pinned on the dress that capture 2-4 fps. The wealth or the hell of life-logging data 22:51
  • 18. Extracting semantics from egocentric images Computer Vision allows to process and analyze huge amount of images and extract the semantics from them. 1822:51 What technology to apply?
  • 22. From NN to CNN 2222:51
  • 24. ConvNets are everywhere 24From: Fei-Fei Li & Andrej Karpathy & Justin Johnson22:51
  • 25. ConvNets are everywhere 25From: Fei-Fei Li & Andrej Karpathy & Justin Johnson22:51
  • 26. ConvNets are everywhere 26From: Fei-Fei Li & Andrej Karpathy & Justin Johnson22:51
  • 27. ConvNets are everywhere 27From: Fei-Fei Li & Andrej Karpathy & Justin Johnson22:51
  • 28. ConvNets are everywhere 28From: Fei-Fei Li & Andrej Karpathy & Justin Johnson22:51
  • 29. Egocentric vision • How can deep learning help process egocentric images? 22:51 29
  • 31. Visual summary through keyframe extraction (joint work UPC & UB) 31 Results 22:51
  • 32. Analysis of human interaction 32 F-formations extraction from egocentric vision 22:51
  • 33. Two main questions? • What we eat? – Automatic food recognition vs. Food diaries • And how we eat? – Automatic eating pattern extraction – when, where, how, how long, with whom, in which context? 3322:51
  • 34. Automatic Food Analysis • Food detection • Food recognition • Food environment recognition • Eating pattern extraction 3422:51
  • 35. Food datasets 35 Food256 - 100images/class Classes: 256 Food101 – 101.000 images Classes: 101 Food101+FoodCAT: 146.392 (101.000+45.392) Classes: 131 EgocentricFood: 5038 images Classes: 9 22:51
  • 36. Food-related environment recognition Food Non Food ... w1 w2 wn G oogleNet Softm ax G AP inception4eoutput Deep Convolution X FAM Bounding Box G eneration 36 Our localization method based on Global Average Pooling (GAP) that produces a Food Activation Map (FAM). Examples of localization and recognition on UECFood256 (top) and EgocentricFood (bottom). Ground truth is shown in green and our method in blue. Marc Bolaños, Petia Radeva: Simultaneous Food Localization and Recognition. submitted to ICPR’16, arXiv.org>cs>arXiv:1604.07953, 2016. 22:51
  • 37. Food environment classification 37 Bakery Banquet hall Bar Butcher shop Cafetería Ice cream parlor Kitchen Kitchenette Market Pantry Picnic Area Restaurant Restaurant Kitchen Restaurant Patio Supermarket Candy store Coffee shop Dinette Dining room Food court Galley Classification results: 0.92 - Food-related vs. Non-food-related 0.68 - 22 classes of Food-related categories22:51
  • 38. Image Input Foodness Map Extraction Food Detection CNN Food Recognition CNN Food Type Recognition Apple Strawberry Food recognition Results: TOP-1 74.7% TOP-5 91.6% SoA (Bossard,2014): TOP-1 56,4% 22:51 38
  • 40. What did I do today? 40 Wearable cameras allows to visualize our diary. Computer Vision allows to process huge amount of data and automatically extracts the semantics from it: where, what, who, how, etc. 22:51
  • 41. Will life-logging and internet of things help know us better? 4122:51
  • 42. How much time I spend by day on different activities? 4222:51
  • 44. With whom I was interacting with? 4422:51
  • 45. How stressful am I in different places? 4522:51
  • 46. “Six Months of My Life” 46by David El Achkar22:51
  • 48. Metabolic diseases and health 4822:51  4.2 million die of chronic diseases in Europe (diabetes or cancer) linked to lack of physical activities and unhealthy diet.  Physical activities can increase lifespan by 1.5-3.7 years.  Obesity is a chronic disease associated with huge economic, social and personal costs.  Risk factors for cancers, cardiovascular and metabolic disorders and leading causes of premature mortality worldwide.
  • 49. Health and medical care • Today, 88% of healthcare costs are spent on medical care – access to physicians, hospitals, procedures, drugs, etc. • However, medical care only accounts for approximately 10% of a person’s health. • Approximately half the decline in U.S. Deaths from coronary heart disease from 1980 through 2000 may be attributable to reductions in major risk factors (systolic blood pressure, smoking, physical inactivity). 4922:51
  • 50. What are we missing in health applications? • Today, automatically measuring physical activity is not a problem. • But what about food and nutrition? – State of the art: Nutritional health apps are based on manual food diaries. 22:51 50 Sparkpeople LoseIt! MyFitnessPal Cronometer Fatsecret
  • 51. Towards healthy habits Towards visualizing summarized lifestyle data to ease the management of the user’s healthy habits (sedentary lifestyles, nutritional activity, etc.). Life-logging can help us to extract our nutritional habits: taking photos of our everyday life and being able to analyse what we eat, starting by the dish recognition, where we eat, with whom we eat, how we eat. 22:51 51
  • 52. Teleassistance and home monitoring Using wearable cameras, we can have a clear picture of the lifestyle the person is having during the whole day: • Is he/she active or passive – doing domestic duties, – performing intellectual activities like reading, – coping with daily functions, etc. 22:51 52
  • 53. Life-logging for cognitive treatment of people with amnesia 53 Program based on life-logging captured by a Wearable Camera recording specific autobiographical episodes for stimulating posteriorly episodic memory function. Using wearable cameras and looking at their autobiographic experiences, people with amnesia improved their memory and cognitive facilities by using episodic images. Claire, a 49-year-old former nurse, six years ago suffered brain damage due to a rare viral infection called herpes encephalitis. Now an amnesiac who is unable to recognize faces, Claire lives in a world in which even her lifelong friends appear as strangers. 22:51
  • 54. Life-logging for MCI treatment 54 Goal: using episodic images to develop cognitive exercises and tools for memory reforcing of MCI and Alzheimer people. To explore the association between changes in cognitive, functional and emotional outcomes. 22:51
  • 55. Application of Lifelogging to Migraine • Objective Biomarkers in Migraine - how to predict a migraine attack • How does the brain interact with the environment - the cues that ensure adaptation 55 Lifelogging can assure individualized tools to detect which are the triggers of the migraine for an individual. 22:51
  • 56. Lifelogging and wearable cameras • Lifelogging technology is here • Computer Vision and Machine Learning can help process big image data • How to get profit of it to help improve our health?! 22:51 56

Editor's Notes

  1. But in fact we are not very interested in every and each of these images but to extract semantic information or actually meanings out of them for later use. These meanings could be anything, from thinking about how I do eat or workout or even communicate with others. Studying these semantic info is an interesting subject since they can help one to observe his own life style or even helping people with memory impairment to improve their memory by reviewing events happened to them during a day. To accomplish this, a search engine is required to use existing clues to “recognize” different events and meanings, this procedure is called life-logging
  2. Other methods also use unsupervised pre-training to structure a neural network, making it first learn generally useful feature detectors. Then the network is trained further by supervised back-propagation to classify labeled data. The deep model of Hinton et al. (2006) involves learning the distribution of a high level representation using successive layers of binary or real-valued latent variables. It uses a restricted Boltzmann machine to model each new layer of higher level features. Each new layer guarantees an increase on the lower-bound of the log likelihood of the data, thus improving the model, if trained properly. Once sufficiently many layers have been learned the deep architecture may be used as a generative model by reproducing the data when sampling down the model (an "ancestral pass") from the top level feature activations.[8] Hinton reports that his models are effective feature extractors over high-dimensional, structured data.[9] Natural Language Processing which is used heavily in language conversion in chat rooms or processing text from where human speeches. Optical Character Recognition which is scanning of images. It's gaining traction lately to read an image and extract text out of it and correlate to the objects found on image Speech Recognition applications like Siri or Cortana needs no introduction Artificial Intelligence induction to different robots for automating at least a minute level of tasks a human can do. We want them to be a little smarter. Drug discovery though medical imaging-based diagnosis using deep learning. It's kind of in early stages now. Check Butterfly Network for the work they are doing. CRM needs for companies are growing day by day. There are hundreds of thousands of companies around the globe from small to big companies who wants to know their potential customers. Deep Learning has provided some outstanding results. Check for companies like RelateIQ (product) who has seen astounding success of using Machine Learning in this area.
  3. Medical applications - there are tremendous advances in robotic surgery that relies on extremely sensitive tactile equipment. However, if a doctor can advise a robot to "move a fraction of a millimeter to the left of the clavicle" they could potentially gain more control by directing the robot via full understood voice control. Automotive - we are already seeing self driving cars; deep learning will possibly integrate into automated driving systems to detect and interpret sights and sounds that might be beyond the capacity of humans. Military - drones are particularly well suited to deep learning. Surveillance - here too drones will play a role, but the idea of computers that are able to sense and interpret with a human-like degree of accuracy will change the way in which surveillance is done.