Exploring the Quality of Health Apps: The ISYS Ranking System

1. 1
iSYS is a Foundation to develop social
health projects in Internet
Exploring the Quality of Health Apps: The ISYS
Ranking System
Medicine 2.0 Malaga, October 2014
Immaculada Grau*, Fundación iSYS, Spain, Group for Research in Primary Care–
IDIBAPS, Barcelona, Spain
José Antonio Gallego Fundación iSYS, Spain,
Francisco J Grajales III, University of British Columbia, Vancouver, Canada
Luis Fernandez-Luque*, Northern Research Institute, Tromso, Norway
Belchin Kostov, Group for Research in Primary Care–IDIBAPS, Barcelona, Spain

2. 2
Why the need for a scoring system in health apps?

3. Introduction
What are the problems…
•The number of mobile Apps on offer is huge, and is growing
exponentially
•Users generally don’t know what they will find until they download
an app and are faced with an overwhelming number of options.
•The trustworthiness of internet-based health information aimed at
non-professional users is an important and widely debated topic.
What do we need…
•Users need security
•Scientific guarantee
•Empowerment through relevant information

4. 4
Objectives
The main objective of the iSYS Ranking is to empirically evaluate the
quality of health-related apps for patients within three
domains:
• (A) Popular interest
• (B) Quality (trust)
• (C) Utility
With this we aim to
1. Promote awareness of different criteria that will be helpful
before downloading a health app.
2. Develop a public ranking of scoring to help developers
improve the quality of their applications and promote
excellence.

5. 5
Objectives
Limitations:
•This method is not a quality or a safety seal. It is only a guide
•Apps to evaluate would be for the general public –not for
healthcare professionals-.
•The Apps evaluated are NOT part of a care process
(diagnosis and monitoring of clinical parameters ARE part of a
care process), unless they were certified by a competent
official body, like the FDA.
•The score is based mainly on the App description provided by
the developer, and the first results of Google search.

6. 6
Methodology
Inspired by: Moher D, Schulz KF, Simera I, Altman DG (2010)
Guidance for Developers of Health Research Reporting Guidelines.
PLoS Med 7(2): e1000217. doi:10.1371/journal.pmed.1000217
1. Needs Assessment
2. Literature Review
3. Obtain Funding for the Rating (Guideline) Initiative
4. Expert Sampling & Delphi Method
5. Present and Discuss Results & prepare face-to-face meeting
6. Develop the Rating System Draft
7. Pilot Testing & Repeat steps 5 & 6
8. Develop an Explanatory Document
9. Publication and Knowledge Translation Strategy

7. 7
Methodology
Literature Review
A broad search string was used to query articles published on the PubMed
database in English and Spanish from 1993 to 2014, without geographic filters.
Results:
•EVALUATION of health applications for Smartphone (23 results)
•ANALYSIS of health applications for Smartphone (18 results)
•QUALITY of health applications for Smartphone (17 results)
A total of 18 articles met our inclusion criteria and their relevant data was
extracted using a standardized template
Outstanding
Bender JL, Yue RYK, To MJ, Deacken L, Jadad AR. A Lot of Action, But Not in the Right Direction:
Systematic Review and Content Analysis of Smartphone Applications for the Prevention,
Detection and Management of Cancer J Med Internet Res 2013;15(12):e287

8. 8
Methodology
4. Identify Participants & Conduct a Delphi Exercise.
Research Team (n=5) Technical team (n=4) Expert panel (n=16)
• Bibliographic review
• Initial recommendations
• Selection of external
Panel of experts
• Prepare 1st Questionnaire
(Q1)
• Send out Q1
• Analysis and evaluation by
the panel
• Prepare and send out 2nd
Questionnaire (Q2)
• Analysis, statistics and
evaluation by the panel
• Results report
• Specific evaluation for
each recommendation
• Extra comments
• Analysis of results
• Possible modification
and forwarding
1st delphi
round
2nd delphi
round
• Proof of Concept
• 5 Apps
• 4 Patients
• 3 MD
• 3 Nurses
• 2 Heath experts
• 1 COM expert
• 3 Tech experts

9. 9
Methodology
On completion of the Delphi method, we carried out an assessment of
five mobile apps. For this evaluation, items and mean scores obtained
from Delphi were used.
Problem
(C) Utility scoring criteria did not work in any case, the research team
decided to take on new criteria, based the literature on review.
These new criteria were initially given the lowest score (3 points on a
scale of 1-5, where 5 was the maximum) obtained during the Delphi
process, pending a new iteration from the panel of experts.
Solution
Both experts scored the five applications in turn and the score worked.
Limitation: These new criteria have not been evaluated by the expert
panel, so the method will be released this year in a beta version.

10. 10
Results
SOURCE
• 11 criteria from Delphi method
• 5 criteria from the new Literature review
A. Popular Interest:
1. Positive Rating given by users who downloaded the App (4
points).
2. Available on 2 or more platforms (3 points)
3. Interest Statement by Patient/Affected Association (4
points).

11. 11
Results
B. Trust / Quality 4. seal from a Health Agency maximum points
(18 points) or
5. Validate by a specialized professional / Public Health Body /
Scientific Society (4 points)
6. Promoted by Affected Association (3 points)
7. The App has a website (responsibility) / Data protection
endorsement (4 points)
8. Cites sources of scientific evidence (4 Points)
9. Names the organization responsible (3 points)

12. 12
Results
C. Utility 10. Wide research group RCT scores maximum (18 points) or
11. Research: No-RCT Small Group [<30 users] (3 points)
12. Declaration of professional scientific society or association /
Partnership Declaration AFFECTED (3 points)
13. Provides information (3 points)
14. Provides info to useful Health Provider / trackers (3 points)
15. Links to other affected / users (3 points)
16. Uses games to promote health data (3 points)

13. 13
Results: example

14. 14
Results: example

15. 15
Conclusions
1. The iSYS method was successful in scoring Health Apps
2. It compliments but does not substitute quality and safety
seals.
3. It’s provides a neutral reference for a patient before they
download an App.

16. 16
Discussion
• We have not found any other objective valuation
methodology to compare Apps.
• We believe this scoring method can make an active
contribution to:
• Empowerment of patients
• Raise awareness of the importance of quality apps
• Encourage excellence in the development of Apps
• Contribute to measurement and development of
standards and guidelines

17. 17
Next steps
• Develop a Web Site.
• Find sponsorship
• Seek and deal with social Feedback and
Criticism (crowd sourcing) to improve upon.
• Update the Method. Periodic release of new
improved versions (Annual?)
• Encourage Endorsement. An annual event to
reward best rated apps, developers and
promoters
• Translate and publish the method.