SlideShare uma empresa Scribd logo
1 de 50
MOBILE TECHNOLOGY
USAGE BY HUMANITARIAN
PROGRAMS:
A METADATA ANALYSIS
Rashmi Dayalu
O P E N
D A T A
S C I E N C E
C O N F E R E N C E_
BOSTON 2015
@opendatasci
1
Mobile technology usage by humanitarian programs:
a metadata analysis
Open Data Science Conference
May 31, 2015
Rashmi Dayalu
Data Scientist
Dimagi, Inc.
2
#ODSC
We are called to work “where our greatest passion
meets the world's greatest need.”
- Frederick Buechner
3
#ODSC
Background
4
Access to humanitarian services is limited in most parts
of the world.
5
#ODSC
6
Frontline Workers (FLWs) are often the primary link
between underserved communities and humanitarian
services.
7
#ODSC
“How can I keep track of my pregnant clients’ medical
information, visit schedules and due dates?” – Seema
8
#ODSC
“How can I show videos of the best agricultural
practices to the farmers in my community?” – Yann
9
Meeting the world’s greatest needs: “We deliver open and innovative technology to
help underserved communities around the world.”
10
#ODSC
Dimagi’s core product
11
#ODSC
 Open source mobile technology platform
 Does not require software developers to configure or deploy mobile
applications
 Can be used on feature phones, androids, tablets, on the web or over
SMS
Image: http://www.ictedge.org/projects/zeprs
X 
12
Data collection Client Counseling
Case management and workflow supportTraining reinforcement and supervision
The result? Stronger healthcare workers
and stronger communities…
#ODSC
13
#ODSC
There are CommCare users in over 40 countries
4001 – 5000
3001 – 4000
2001 – 3000
1001 – 2000
1 - 1000
14
#ODSC
CommCare is used across multiple sectors and subsectors
15
#ODSC
• CommCare’s cloud server hosts data from hundreds of
humanitarian programs.
• We are using CommCare metadata to ask a variety of questions
that can aid programs and FLWs in their goals.
http://noble1solutions.com/wp-content/uploads/2014/06/what-is-big-data.jpg
16
#ODSC
BREADTH
D
E
P
T
H
Program A Program B …. Program “X”
FLW 1
FLW 2
FLW 3
…
FLW “N”
How do programs and FLWs perform across the board?
Ismyprogramperformingwell?
AremyFLWsperformingwell?
17
Measuring FLW and program performance
using CommCare metadata
18
Form creation is the most basic unit of CommCare usage
#ODSC
19
CommCare form metadata
FLW ID CASE ID
Form start/end date
Form start/end time
#ODSC
20
#ODSC
Dimagi’s data platform
21
#ODSC
Cumulative # form submissions and # new cases registered with Commcare:
all programs, 2010 - 2014
22
#ODSC
1. How quickly do FLWs develop proficiency
with CommCare?
Analysis by: Jeremy Wacksman
23
#ODSC
We looked at 634 workers who used CommCare for at least one year and
were active for at least 10 months of their first year.
Q1
Q2
Quarterly
range
Median
change
Q1 – Q2 + 22.9%
Q2 – Q3 + 1.9%
Q3 – Q4 + 0.0%
Q3
Q4
24
#ODSC
2. Are FLW activity levels stable from month to
month?
25
#ODSC
Intra-user consistency:
• After the 6 month adoption period, do FLWs
maintain stable levels of CommCare activity?
• We calculated the Pearson correlation coefficient for
all pairs of consecutive calendar months for
individual FLW activity levels
26
#ODSC
Intra-user consistency for % active days and # forms (663 FLWs)
r = 0.68r = 0.70
27
#ODSC
Programs can use the hypothesis of intra-user consistency to
monitor unexpected changes in FLW activity levels:
e.g. (1) Are FLWs less active during certain seasons or
months of the year?
N = 5,303 monthly observations (from health programs in India)
28
#ODSC
(2) Do FLWs show decreased activity levels prior to
attrition in CommCare activity (inactivity >= 90 days)?
N = 252 FLWs with at least one CommCare attrition event
29
#ODSC
3. Do FLW activity levels follow a bell curve?
Analysis by: Mengji Chen
30
#ODSC
1. Boyle, E., Aguinis, H. “The Best & the Rest: Revisiting the Norm of Normality and Individual Performance”, Personnel
Psychology, 2012, 65, 79-119.
2. Image from: http://www.marin.edu/~npsomas/Lectures/Ch_1/Section_03.htm
Normal distributions are the most commonly held assumption in performance
metrics1. Is this assumption valid for CommCare FLWs?
31
#ODSC
Distribution of % active days/month for 12 programs with the most active months
32
#ODSC
Boyle, 2012. Personnel Psychology
Programs have larger number of FLWs that are either underperforming or hyper-
performing.
Workloads, performance ranking, training and compensation cannot assume the
norm of normality.
33
#ODSC
4. Do FLWs use CommCare in real-time while
interacting with their clients?
34
#ODSC
There is no way to confirm real-time data collection by FLWs using
metadata, but we can flag visit data that was unlikely to have been
entered in real-time:
1. Batch entry – visits entered consecutively in quick succession (e.g.
with < 10 minutes between visits)
2. Visit duration (e.g. < 1 minute)
3. Visit time of day (e.g. visits started at night, between 6pm – 6am)
35
#ODSC
Batch visits (%) by program
Proportion of batch visits from 30 maternal and child health programs worldwide
36
#ODSC
Programs with unexpectedly large daily visit volumes revealed that:
(1) Patient data was often uploaded automatically via CommCareHQ -
CommCare’s web interface (e.g. maternal registrations)
37
#ODSC
(2) Manual batch entry might actually be part of regular work flow for FLWs in
clinical settings (e.g. immunizations, child anthropometrics, etc.)
We looked at batch entry rates for 9 programs that had at least one “travel visit”
component built into their apps.
38
#ODSC
In conjunction with batch entry, visit duration and visit time of day can be used to
flag visit data that was unlikely collected in real-time.
Visit duration (Mood’s Median Test):
Batch visits for programs A, B and C combined were ~half the duration of non-batch
visits (median duration of batch visits = 3.8 minutes, median duration of non-batch
visits = 7.7 minutes, Z = 5.35, p < 0.001).
Visit time of day (Chi-square Test):
Batch were more likely to have been recorded in the night
(% night non-batch visits = 16.5% and % night batch visits = 20.8%, 2 = 178.99, df = 1, p
<0.001).
39
#ODSC
5. How long do programs use CommCare for?
Analysis by: Patrick Keating
40
#ODSC
Sustainable use of CommCare is evidence for CommCare’s value. Of 306 programs,
how many were still active in Q4 2014?
Distribution of # programs by # active months and activity status in Q4 2014
41
#ODSC
176 (57.5%) programs stopped using CommCare for at least 3 months. Of those,
43% restarted their CommCare usage, though restart rates are dependent on the
age of the program.
42
#ODSC
Programs with more active FLWs were more likely to be active through 2014
This could mean that programs with smaller numbers of users have limited
resources and sometimes cannot continue their activities - regardless of how
effective CommCare is.
43
#ODSC
6. Which programs are improving over time?
Algorithm developed by:
Dag Holmboe
Dimagi’s Data Science Advisor
Founder of Klurig Analytics
http://www.kluriganalytics.com
44
#ODSC
Detecting improvement can help us concretely identify the programmatic factors
that led to the improvement.
45
#ODSC
Preliminary validation: Program #60
Performance feedback to FLWs in the middle of the year could have
contributed to the continued improvement (beyond first 6 months).
46
#ODSC
Some future investigations:
1. Do 20% of FLWs submit 80% of the data?
2. Do programs that use supervisory tools have
the most active FLWS?
3. Is CommCare activity correlated with socio-
economic indicators (GNP, literacy rates,
corruption index, etc.)?
4. How do CommCare crashes affect user
behavior?
47
Thank you!
For questions or research opportunities, please contact:
Rashmi Dayalu
rdayalu@dimagi.com
#ODSC
48
#ODSC
Activity metric by FLW
per calendar month
Definition
1. # forms Total number of electronic forms submitted
2. # visits Total number of visits made to all cases
3. # cases Total number of unique cases visited (either registered
or followed up)
4. # cases registered Total number of unique cases registered
5. # cases followed-up Total number of unique cases followed-up
6. % of active days Percentage of days in the month during which the CHW
submitted data
7. Total duration of
CommCare use (min)
Cumulative time using CommCare, i.e. sum of all visit
durations
CommCare activity metrics - Aggregated by calendar month
per FLW
49
#ODSC
Calculating breakpoints:
• Rolling window of mean FLW activity levels over the program
lifetime
• If window mean is at least 3 SD’s higher then the previous window, it
is a candidate breakpoint
• t-test of means between the windows confirms the statistical
significance of the breakpoint

Mais conteúdo relacionado

Destaque

CommCare Workshop_Javetski and Wacksmon_4.22.13
CommCare Workshop_Javetski and Wacksmon_4.22.13CommCare Workshop_Javetski and Wacksmon_4.22.13
CommCare Workshop_Javetski and Wacksmon_4.22.13
CORE Group
 
M health_Farhad
M health_FarhadM health_Farhad
M health_Farhad
Farhad Ali
 

Destaque (13)

CommCare Workshop_Javetski and Wacksmon_4.22.13
CommCare Workshop_Javetski and Wacksmon_4.22.13CommCare Workshop_Javetski and Wacksmon_4.22.13
CommCare Workshop_Javetski and Wacksmon_4.22.13
 
Training on using kobotoolbox for on line or off-line
Training on using kobotoolbox for on line or off-lineTraining on using kobotoolbox for on line or off-line
Training on using kobotoolbox for on line or off-line
 
Open your data with CartoDB
Open your data with CartoDBOpen your data with CartoDB
Open your data with CartoDB
 
Capstone.
Capstone.Capstone.
Capstone.
 
Machine-In-The-Loop for Knowledge Discovery
Machine-In-The-Loop for Knowledge DiscoveryMachine-In-The-Loop for Knowledge Discovery
Machine-In-The-Loop for Knowledge Discovery
 
Spark, Python and Parquet
Spark, Python and Parquet Spark, Python and Parquet
Spark, Python and Parquet
 
M health_Farhad
M health_FarhadM health_Farhad
M health_Farhad
 
PPT_PME_Dhital B N
PPT_PME_Dhital B NPPT_PME_Dhital B N
PPT_PME_Dhital B N
 
Perspectivas profesionales en geomática 2014
Perspectivas profesionales en geomática 2014Perspectivas profesionales en geomática 2014
Perspectivas profesionales en geomática 2014
 
Von der Businessstrategie zum Service-Portfolio
Von der Businessstrategie zum Service-PortfolioVon der Businessstrategie zum Service-Portfolio
Von der Businessstrategie zum Service-Portfolio
 
Open Source Tools & Data Science Competitions
Open Source Tools & Data Science Competitions Open Source Tools & Data Science Competitions
Open Source Tools & Data Science Competitions
 
CARTO BUILDER: from visualization to geospatial analysis
CARTO BUILDER: from visualization to geospatial analysisCARTO BUILDER: from visualization to geospatial analysis
CARTO BUILDER: from visualization to geospatial analysis
 
Tidal scale short_story_v2
Tidal scale short_story_v2Tidal scale short_story_v2
Tidal scale short_story_v2
 

Semelhante a Mobile technology Usage by Humanitarian Programs: A Metadata Analysis

Knowledge Matters Issue 15 - Technology at Concern
Knowledge Matters Issue 15 - Technology at ConcernKnowledge Matters Issue 15 - Technology at Concern
Knowledge Matters Issue 15 - Technology at Concern
Ellen Ward
 
FISHBONE ANALYSIS ON WASTES IN SOFTWARE DEVELOPMENT USING THE LEAN I.T. PRINC...
FISHBONE ANALYSIS ON WASTES IN SOFTWARE DEVELOPMENT USING THE LEAN I.T. PRINC...FISHBONE ANALYSIS ON WASTES IN SOFTWARE DEVELOPMENT USING THE LEAN I.T. PRINC...
FISHBONE ANALYSIS ON WASTES IN SOFTWARE DEVELOPMENT USING THE LEAN I.T. PRINC...
ecij
 
Presentation on DR testing featuring quotes by Robert Nardella in an intervie...
Presentation on DR testing featuring quotes by Robert Nardella in an intervie...Presentation on DR testing featuring quotes by Robert Nardella in an intervie...
Presentation on DR testing featuring quotes by Robert Nardella in an intervie...
Robert Nardella
 
Mobile CRM Webinar: 6 Steps to Maximize Mobile ROI for Government Agencies
Mobile CRM Webinar: 6 Steps to Maximize Mobile ROI for Government AgenciesMobile CRM Webinar: 6 Steps to Maximize Mobile ROI for Government Agencies
Mobile CRM Webinar: 6 Steps to Maximize Mobile ROI for Government Agencies
Waterfall
 
Copyright © 2015-2018 VIT, All Rights Reserved. .docx
Copyright © 2015-2018 VIT, All Rights Reserved. .docxCopyright © 2015-2018 VIT, All Rights Reserved. .docx
Copyright © 2015-2018 VIT, All Rights Reserved. .docx
dickonsondorris
 

Semelhante a Mobile technology Usage by Humanitarian Programs: A Metadata Analysis (20)

Prosci's "CMROI Calculator: Interactive Inside Look" Webinar
Prosci's "CMROI Calculator: Interactive Inside Look" WebinarProsci's "CMROI Calculator: Interactive Inside Look" Webinar
Prosci's "CMROI Calculator: Interactive Inside Look" Webinar
 
Knowledge Matters Issue 15 - Technology at Concern
Knowledge Matters Issue 15 - Technology at ConcernKnowledge Matters Issue 15 - Technology at Concern
Knowledge Matters Issue 15 - Technology at Concern
 
FISHBONE ANALYSIS ON WASTES IN SOFTWARE DEVELOPMENT USING THE LEAN I.T. PRINC...
FISHBONE ANALYSIS ON WASTES IN SOFTWARE DEVELOPMENT USING THE LEAN I.T. PRINC...FISHBONE ANALYSIS ON WASTES IN SOFTWARE DEVELOPMENT USING THE LEAN I.T. PRINC...
FISHBONE ANALYSIS ON WASTES IN SOFTWARE DEVELOPMENT USING THE LEAN I.T. PRINC...
 
IRJET- Trend Analysis on Twitter
IRJET- Trend Analysis on TwitterIRJET- Trend Analysis on Twitter
IRJET- Trend Analysis on Twitter
 
COMMUNICATION, COLLABORATION, AND TEAMWORK ARE INCREASINGLY IMPORTANT FOR THE...
COMMUNICATION, COLLABORATION, AND TEAMWORK ARE INCREASINGLY IMPORTANT FOR THE...COMMUNICATION, COLLABORATION, AND TEAMWORK ARE INCREASINGLY IMPORTANT FOR THE...
COMMUNICATION, COLLABORATION, AND TEAMWORK ARE INCREASINGLY IMPORTANT FOR THE...
 
Believe it or not - keynote CAS 2015
Believe it or not - keynote CAS 2015Believe it or not - keynote CAS 2015
Believe it or not - keynote CAS 2015
 
Maximizing Virtual Productivity in Times of COVID.pptx
Maximizing Virtual Productivity in Times of COVID.pptxMaximizing Virtual Productivity in Times of COVID.pptx
Maximizing Virtual Productivity in Times of COVID.pptx
 
Open Enrollment 2020: Case Study and Trends Report
Open Enrollment 2020: Case Study and Trends ReportOpen Enrollment 2020: Case Study and Trends Report
Open Enrollment 2020: Case Study and Trends Report
 
Business case for deploying online collaboration across organisational bounda...
Business case for deploying online collaboration across organisational bounda...Business case for deploying online collaboration across organisational bounda...
Business case for deploying online collaboration across organisational bounda...
 
Productivity Factors in Software Development for PC Platform
Productivity Factors in Software Development for PC PlatformProductivity Factors in Software Development for PC Platform
Productivity Factors in Software Development for PC Platform
 
5 Top Jargons That Will Help You Make a Big Social Impact Presentation
5 Top Jargons That Will Help You Make a Big Social Impact Presentation5 Top Jargons That Will Help You Make a Big Social Impact Presentation
5 Top Jargons That Will Help You Make a Big Social Impact Presentation
 
Presentation on DR testing featuring quotes by Robert Nardella in an intervie...
Presentation on DR testing featuring quotes by Robert Nardella in an intervie...Presentation on DR testing featuring quotes by Robert Nardella in an intervie...
Presentation on DR testing featuring quotes by Robert Nardella in an intervie...
 
Borys Pratsiuk "How to be NVidia partner"
Borys Pratsiuk "How to be NVidia partner"Borys Pratsiuk "How to be NVidia partner"
Borys Pratsiuk "How to be NVidia partner"
 
Apperian 2015 Executive Enterprise Mobility Survey
Apperian 2015 Executive Enterprise Mobility SurveyApperian 2015 Executive Enterprise Mobility Survey
Apperian 2015 Executive Enterprise Mobility Survey
 
The Critical Role of Spatial Data in Today's Data Ecosystem
The Critical Role of Spatial Data in Today's Data EcosystemThe Critical Role of Spatial Data in Today's Data Ecosystem
The Critical Role of Spatial Data in Today's Data Ecosystem
 
Webinar: Building a Business Case for Enterprise Search
Webinar: Building a Business Case for Enterprise SearchWebinar: Building a Business Case for Enterprise Search
Webinar: Building a Business Case for Enterprise Search
 
Mobile CRM Webinar: 6 Steps to Maximize Mobile ROI for Government Agencies
Mobile CRM Webinar: 6 Steps to Maximize Mobile ROI for Government AgenciesMobile CRM Webinar: 6 Steps to Maximize Mobile ROI for Government Agencies
Mobile CRM Webinar: 6 Steps to Maximize Mobile ROI for Government Agencies
 
Copyright © 2015-2018 VIT, All Rights Reserved. .docx
Copyright © 2015-2018 VIT, All Rights Reserved. .docxCopyright © 2015-2018 VIT, All Rights Reserved. .docx
Copyright © 2015-2018 VIT, All Rights Reserved. .docx
 
Lfa
LfaLfa
Lfa
 
Fdrg bcn-014 - m icf progress report (stefanus snyman & olaf kraus de camargo)
Fdrg bcn-014 - m icf progress report (stefanus snyman & olaf kraus de camargo)Fdrg bcn-014 - m icf progress report (stefanus snyman & olaf kraus de camargo)
Fdrg bcn-014 - m icf progress report (stefanus snyman & olaf kraus de camargo)
 

Mais de odsc

Kaggle The Home of Data Science
Kaggle The Home of Data ScienceKaggle The Home of Data Science
Kaggle The Home of Data Science
odsc
 
The Art of Data Science
The Art of Data Science The Art of Data Science
The Art of Data Science
odsc
 
Frontiers of Open Data Science Research
Frontiers of Open Data Science ResearchFrontiers of Open Data Science Research
Frontiers of Open Data Science Research
odsc
 

Mais de odsc (20)

Big Data Infrastructure: Introduction to Hadoop with MapReduce, Pig, and Hive
Big Data Infrastructure: Introduction to Hadoop with MapReduce, Pig, and HiveBig Data Infrastructure: Introduction to Hadoop with MapReduce, Pig, and Hive
Big Data Infrastructure: Introduction to Hadoop with MapReduce, Pig, and Hive
 
Think Breadth, Not Depth
Think Breadth, Not DepthThink Breadth, Not Depth
Think Breadth, Not Depth
 
Data Science at Dow Jones: Monetizing Data, News and Information
Data Science at Dow Jones: Monetizing Data, News and InformationData Science at Dow Jones: Monetizing Data, News and Information
Data Science at Dow Jones: Monetizing Data, News and Information
 
Building a Predictive Analytics Solution with Azure ML
Building a Predictive Analytics Solution with Azure MLBuilding a Predictive Analytics Solution with Azure ML
Building a Predictive Analytics Solution with Azure ML
 
Beyond Names
Beyond NamesBeyond Names
Beyond Names
 
How Woman are Conquering the S&P 500
How Woman are Conquering the S&P 500How Woman are Conquering the S&P 500
How Woman are Conquering the S&P 500
 
Domain Expertise and Unstructured Data
Domain Expertise and Unstructured DataDomain Expertise and Unstructured Data
Domain Expertise and Unstructured Data
 
Kaggle The Home of Data Science
Kaggle The Home of Data ScienceKaggle The Home of Data Science
Kaggle The Home of Data Science
 
Machine Learning with scikit-learn
Machine Learning with scikit-learnMachine Learning with scikit-learn
Machine Learning with scikit-learn
 
Bridging the Gap Between Data and Insight using Open-Source Tools
Bridging the Gap Between Data and Insight using Open-Source ToolsBridging the Gap Between Data and Insight using Open-Source Tools
Bridging the Gap Between Data and Insight using Open-Source Tools
 
Top 10 Signs of the Textpocalypse
Top 10 Signs of the TextpocalypseTop 10 Signs of the Textpocalypse
Top 10 Signs of the Textpocalypse
 
The Art of Data Science
The Art of Data Science The Art of Data Science
The Art of Data Science
 
Frontiers of Open Data Science Research
Frontiers of Open Data Science ResearchFrontiers of Open Data Science Research
Frontiers of Open Data Science Research
 
Feature Engineering
Feature Engineering Feature Engineering
Feature Engineering
 
Agile Data
Agile DataAgile Data
Agile Data
 
Using your powers for good: Data science in the social sector
Using your powers for good: Data science in the social sectorUsing your powers for good: Data science in the social sector
Using your powers for good: Data science in the social sector
 
Machine Learning for Suits
Machine Learning for SuitsMachine Learning for Suits
Machine Learning for Suits
 
Recurrent Neural Networks for Text Analysis
Recurrent Neural Networks for Text AnalysisRecurrent Neural Networks for Text Analysis
Recurrent Neural Networks for Text Analysis
 
Predictive Modeling Workshop
Predictive Modeling WorkshopPredictive Modeling Workshop
Predictive Modeling Workshop
 
Enabling Graph Analytics at Scale: The Opportunity for GPU-Acceleration of D...
Enabling Graph Analytics at Scale:  The Opportunity for GPU-Acceleration of D...Enabling Graph Analytics at Scale:  The Opportunity for GPU-Acceleration of D...
Enabling Graph Analytics at Scale: The Opportunity for GPU-Acceleration of D...
 

Último

Artificial Intelligence: Facts and Myths
Artificial Intelligence: Facts and MythsArtificial Intelligence: Facts and Myths
Artificial Intelligence: Facts and Myths
Joaquim Jorge
 
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
?#DUbAI#??##{{(☎️+971_581248768%)**%*]'#abortion pills for sale in dubai@
 

Último (20)

AWS Community Day CPH - Three problems of Terraform
AWS Community Day CPH - Three problems of TerraformAWS Community Day CPH - Three problems of Terraform
AWS Community Day CPH - Three problems of Terraform
 
Boost PC performance: How more available memory can improve productivity
Boost PC performance: How more available memory can improve productivityBoost PC performance: How more available memory can improve productivity
Boost PC performance: How more available memory can improve productivity
 
TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
TrustArc Webinar - Stay Ahead of US State Data Privacy Law DevelopmentsTrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
 
Tech Trends Report 2024 Future Today Institute.pdf
Tech Trends Report 2024 Future Today Institute.pdfTech Trends Report 2024 Future Today Institute.pdf
Tech Trends Report 2024 Future Today Institute.pdf
 
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
 
Developing An App To Navigate The Roads of Brazil
Developing An App To Navigate The Roads of BrazilDeveloping An App To Navigate The Roads of Brazil
Developing An App To Navigate The Roads of Brazil
 
Real Time Object Detection Using Open CV
Real Time Object Detection Using Open CVReal Time Object Detection Using Open CV
Real Time Object Detection Using Open CV
 
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, AdobeApidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
 
GenAI Risks & Security Meetup 01052024.pdf
GenAI Risks & Security Meetup 01052024.pdfGenAI Risks & Security Meetup 01052024.pdf
GenAI Risks & Security Meetup 01052024.pdf
 
04-2024-HHUG-Sales-and-Marketing-Alignment.pptx
04-2024-HHUG-Sales-and-Marketing-Alignment.pptx04-2024-HHUG-Sales-and-Marketing-Alignment.pptx
04-2024-HHUG-Sales-and-Marketing-Alignment.pptx
 
The 7 Things I Know About Cyber Security After 25 Years | April 2024
The 7 Things I Know About Cyber Security After 25 Years | April 2024The 7 Things I Know About Cyber Security After 25 Years | April 2024
The 7 Things I Know About Cyber Security After 25 Years | April 2024
 
A Year of the Servo Reboot: Where Are We Now?
A Year of the Servo Reboot: Where Are We Now?A Year of the Servo Reboot: Where Are We Now?
A Year of the Servo Reboot: Where Are We Now?
 
How to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerHow to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected Worker
 
Artificial Intelligence: Facts and Myths
Artificial Intelligence: Facts and MythsArtificial Intelligence: Facts and Myths
Artificial Intelligence: Facts and Myths
 
Advantages of Hiring UIUX Design Service Providers for Your Business
Advantages of Hiring UIUX Design Service Providers for Your BusinessAdvantages of Hiring UIUX Design Service Providers for Your Business
Advantages of Hiring UIUX Design Service Providers for Your Business
 
Understanding Discord NSFW Servers A Guide for Responsible Users.pdf
Understanding Discord NSFW Servers A Guide for Responsible Users.pdfUnderstanding Discord NSFW Servers A Guide for Responsible Users.pdf
Understanding Discord NSFW Servers A Guide for Responsible Users.pdf
 
What Are The Drone Anti-jamming Systems Technology?
What Are The Drone Anti-jamming Systems Technology?What Are The Drone Anti-jamming Systems Technology?
What Are The Drone Anti-jamming Systems Technology?
 
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
 
Powerful Google developer tools for immediate impact! (2023-24 C)
Powerful Google developer tools for immediate impact! (2023-24 C)Powerful Google developer tools for immediate impact! (2023-24 C)
Powerful Google developer tools for immediate impact! (2023-24 C)
 
Tata AIG General Insurance Company - Insurer Innovation Award 2024
Tata AIG General Insurance Company - Insurer Innovation Award 2024Tata AIG General Insurance Company - Insurer Innovation Award 2024
Tata AIG General Insurance Company - Insurer Innovation Award 2024
 

Mobile technology Usage by Humanitarian Programs: A Metadata Analysis

  • 1. MOBILE TECHNOLOGY USAGE BY HUMANITARIAN PROGRAMS: A METADATA ANALYSIS Rashmi Dayalu O P E N D A T A S C I E N C E C O N F E R E N C E_ BOSTON 2015 @opendatasci
  • 2. 1 Mobile technology usage by humanitarian programs: a metadata analysis Open Data Science Conference May 31, 2015 Rashmi Dayalu Data Scientist Dimagi, Inc.
  • 3. 2 #ODSC We are called to work “where our greatest passion meets the world's greatest need.” - Frederick Buechner
  • 5. 4 Access to humanitarian services is limited in most parts of the world.
  • 7. 6 Frontline Workers (FLWs) are often the primary link between underserved communities and humanitarian services.
  • 8. 7 #ODSC “How can I keep track of my pregnant clients’ medical information, visit schedules and due dates?” – Seema
  • 9. 8 #ODSC “How can I show videos of the best agricultural practices to the farmers in my community?” – Yann
  • 10. 9 Meeting the world’s greatest needs: “We deliver open and innovative technology to help underserved communities around the world.”
  • 12. 11 #ODSC  Open source mobile technology platform  Does not require software developers to configure or deploy mobile applications  Can be used on feature phones, androids, tablets, on the web or over SMS Image: http://www.ictedge.org/projects/zeprs X 
  • 13. 12 Data collection Client Counseling Case management and workflow supportTraining reinforcement and supervision The result? Stronger healthcare workers and stronger communities… #ODSC
  • 14. 13 #ODSC There are CommCare users in over 40 countries 4001 – 5000 3001 – 4000 2001 – 3000 1001 – 2000 1 - 1000
  • 15. 14 #ODSC CommCare is used across multiple sectors and subsectors
  • 16. 15 #ODSC • CommCare’s cloud server hosts data from hundreds of humanitarian programs. • We are using CommCare metadata to ask a variety of questions that can aid programs and FLWs in their goals. http://noble1solutions.com/wp-content/uploads/2014/06/what-is-big-data.jpg
  • 17. 16 #ODSC BREADTH D E P T H Program A Program B …. Program “X” FLW 1 FLW 2 FLW 3 … FLW “N” How do programs and FLWs perform across the board? Ismyprogramperformingwell? AremyFLWsperformingwell?
  • 18. 17 Measuring FLW and program performance using CommCare metadata
  • 19. 18 Form creation is the most basic unit of CommCare usage #ODSC
  • 20. 19 CommCare form metadata FLW ID CASE ID Form start/end date Form start/end time #ODSC
  • 22. 21 #ODSC Cumulative # form submissions and # new cases registered with Commcare: all programs, 2010 - 2014
  • 23. 22 #ODSC 1. How quickly do FLWs develop proficiency with CommCare? Analysis by: Jeremy Wacksman
  • 24. 23 #ODSC We looked at 634 workers who used CommCare for at least one year and were active for at least 10 months of their first year. Q1 Q2 Quarterly range Median change Q1 – Q2 + 22.9% Q2 – Q3 + 1.9% Q3 – Q4 + 0.0% Q3 Q4
  • 25. 24 #ODSC 2. Are FLW activity levels stable from month to month?
  • 26. 25 #ODSC Intra-user consistency: • After the 6 month adoption period, do FLWs maintain stable levels of CommCare activity? • We calculated the Pearson correlation coefficient for all pairs of consecutive calendar months for individual FLW activity levels
  • 27. 26 #ODSC Intra-user consistency for % active days and # forms (663 FLWs) r = 0.68r = 0.70
  • 28. 27 #ODSC Programs can use the hypothesis of intra-user consistency to monitor unexpected changes in FLW activity levels: e.g. (1) Are FLWs less active during certain seasons or months of the year? N = 5,303 monthly observations (from health programs in India)
  • 29. 28 #ODSC (2) Do FLWs show decreased activity levels prior to attrition in CommCare activity (inactivity >= 90 days)? N = 252 FLWs with at least one CommCare attrition event
  • 30. 29 #ODSC 3. Do FLW activity levels follow a bell curve? Analysis by: Mengji Chen
  • 31. 30 #ODSC 1. Boyle, E., Aguinis, H. “The Best & the Rest: Revisiting the Norm of Normality and Individual Performance”, Personnel Psychology, 2012, 65, 79-119. 2. Image from: http://www.marin.edu/~npsomas/Lectures/Ch_1/Section_03.htm Normal distributions are the most commonly held assumption in performance metrics1. Is this assumption valid for CommCare FLWs?
  • 32. 31 #ODSC Distribution of % active days/month for 12 programs with the most active months
  • 33. 32 #ODSC Boyle, 2012. Personnel Psychology Programs have larger number of FLWs that are either underperforming or hyper- performing. Workloads, performance ranking, training and compensation cannot assume the norm of normality.
  • 34. 33 #ODSC 4. Do FLWs use CommCare in real-time while interacting with their clients?
  • 35. 34 #ODSC There is no way to confirm real-time data collection by FLWs using metadata, but we can flag visit data that was unlikely to have been entered in real-time: 1. Batch entry – visits entered consecutively in quick succession (e.g. with < 10 minutes between visits) 2. Visit duration (e.g. < 1 minute) 3. Visit time of day (e.g. visits started at night, between 6pm – 6am)
  • 36. 35 #ODSC Batch visits (%) by program Proportion of batch visits from 30 maternal and child health programs worldwide
  • 37. 36 #ODSC Programs with unexpectedly large daily visit volumes revealed that: (1) Patient data was often uploaded automatically via CommCareHQ - CommCare’s web interface (e.g. maternal registrations)
  • 38. 37 #ODSC (2) Manual batch entry might actually be part of regular work flow for FLWs in clinical settings (e.g. immunizations, child anthropometrics, etc.) We looked at batch entry rates for 9 programs that had at least one “travel visit” component built into their apps.
  • 39. 38 #ODSC In conjunction with batch entry, visit duration and visit time of day can be used to flag visit data that was unlikely collected in real-time. Visit duration (Mood’s Median Test): Batch visits for programs A, B and C combined were ~half the duration of non-batch visits (median duration of batch visits = 3.8 minutes, median duration of non-batch visits = 7.7 minutes, Z = 5.35, p < 0.001). Visit time of day (Chi-square Test): Batch were more likely to have been recorded in the night (% night non-batch visits = 16.5% and % night batch visits = 20.8%, 2 = 178.99, df = 1, p <0.001).
  • 40. 39 #ODSC 5. How long do programs use CommCare for? Analysis by: Patrick Keating
  • 41. 40 #ODSC Sustainable use of CommCare is evidence for CommCare’s value. Of 306 programs, how many were still active in Q4 2014? Distribution of # programs by # active months and activity status in Q4 2014
  • 42. 41 #ODSC 176 (57.5%) programs stopped using CommCare for at least 3 months. Of those, 43% restarted their CommCare usage, though restart rates are dependent on the age of the program.
  • 43. 42 #ODSC Programs with more active FLWs were more likely to be active through 2014 This could mean that programs with smaller numbers of users have limited resources and sometimes cannot continue their activities - regardless of how effective CommCare is.
  • 44. 43 #ODSC 6. Which programs are improving over time? Algorithm developed by: Dag Holmboe Dimagi’s Data Science Advisor Founder of Klurig Analytics http://www.kluriganalytics.com
  • 45. 44 #ODSC Detecting improvement can help us concretely identify the programmatic factors that led to the improvement.
  • 46. 45 #ODSC Preliminary validation: Program #60 Performance feedback to FLWs in the middle of the year could have contributed to the continued improvement (beyond first 6 months).
  • 47. 46 #ODSC Some future investigations: 1. Do 20% of FLWs submit 80% of the data? 2. Do programs that use supervisory tools have the most active FLWS? 3. Is CommCare activity correlated with socio- economic indicators (GNP, literacy rates, corruption index, etc.)? 4. How do CommCare crashes affect user behavior?
  • 48. 47 Thank you! For questions or research opportunities, please contact: Rashmi Dayalu rdayalu@dimagi.com #ODSC
  • 49. 48 #ODSC Activity metric by FLW per calendar month Definition 1. # forms Total number of electronic forms submitted 2. # visits Total number of visits made to all cases 3. # cases Total number of unique cases visited (either registered or followed up) 4. # cases registered Total number of unique cases registered 5. # cases followed-up Total number of unique cases followed-up 6. % of active days Percentage of days in the month during which the CHW submitted data 7. Total duration of CommCare use (min) Cumulative time using CommCare, i.e. sum of all visit durations CommCare activity metrics - Aggregated by calendar month per FLW
  • 50. 49 #ODSC Calculating breakpoints: • Rolling window of mean FLW activity levels over the program lifetime • If window mean is at least 3 SD’s higher then the previous window, it is a candidate breakpoint • t-test of means between the windows confirms the statistical significance of the breakpoint

Notas do Editor

  1. Open source Supports longitudinal tracking Designed for low-literate users Runs on Java & Android Runs offline Supports SMS Has an app builder designed for non-programmers Last year we had 37 self starter programs!