2. Nippak et al. 121
it appears that the planning and selection process used to determine the key indicators within the study
can aid in the development of a balanced scorecard for a health information management department. In
addition, it is important to include the health information management department staff in all stages of the
balanced scorecard development, implementation, and evaluation phases.
Keywords
healthcare service innovation and information technology, information and knowledge management,
information technology design and development methodologies, information technology healthcare
evaluation, organizational change and information technology
Introduction
In an effort to more effectively manage healthcare resources, the balanced scorecard (BSC) has
been adopted from the business sector where it has been used for decades as a strategic manage-
ment tool to address issues like operational efficiency and performance management.1,2
BSCs improve internal capacity and support organizational development.3 As such, managers
have used the BSC to initiate operational discussions and other activities such as information
dissemination, process reporting, and knowledge exchange with staff.4 Several benefits of BSCs
have been identified within the literature, which include providing a simple means of translating
an organization’s strategy into operational terms, aligning the organization’s strategy with its
structure, facilitating the development of a process to help communicate the organization’s strat-
egy, and supporting the continuous transformation of the organization’s vision.5 When properly
designed and correctly understood, the BSC has been known to effectively communicate desired
results, provide feedback, facilitate learning, increase accountability, and enhance employee
motivation.6 Because of the overall effectiveness of the BSC as a management tool,7 more not-
for-profit organizations and government agencies have employed a BSC in their strategic man-
agement practice.
Kaplan and Norton8 developed the BSC as a strategic framework in an effort to integrate perfor-
mance indicators derived from organizational strategy. Since its development, BSCs have been the
focus of many research studies particularly in the area of strategic management5 and performance
measurement.9 The original design of the BSC consisted of four perspectives or dimensions: (1)
financial, (2) customer, (3) internal processes, and (4) growth and learning.5 Financial indicators
measure whether an organization’s strategic initiatives are contributing to bottom-line improve-
ment. Performance measures from the customer perspective look at whether the organization cre-
ates value through customers. Internal process indicators identify the critical internal processes
required to help the organization excel. Indicators in the growth and learning perspective such as
people, systems, and procedures help identify essential infrastructure that the organization must
build to create long-term growth and improvement.
To design an effective BSC, three structural attributes need to be considered. According to
Kaplan and Norton,1,10 the BSC metrics must measure those activities that lead to the implementa-
tion of the strategy and should focus on strategic initiatives derived from the organization’s strat-
egy. More recently, Soderberg et al.6 stressed that performance measures are a minimum criteria for
the tool to be considered to be a BSC. As well, another key factor is that balance exists between the
four BSC domains. Most importantly, this balance should exist between outcome and driver meas-
ures, and the financial and non-financial indicators. Finally, both measures need to be causally
linked. A well-designed BSC contains linkages between different indicators within each perfor-
mance dimension and also across the BSC dimensions.6
3. 122 Health Informatics Journal 22(2)
In terms of utilization of BSCs, in 2000, 43 Canadian hospitals, which represent less than 3
percent of all Canadian hospitals, reported using a BSC as a management tool in their organiza-
tion.11 About three-quarters of those surveyed reported successful implementation of the BSC.
Furthermore, administrators who implemented a BSC in these Canadian hospitals forecasted that
the utilization of a BSC would change significantly over the following 5 years of implementation.11
Because the BSC can point to issues that can be addressed by management, the interest of health-
care organizations in employing the BSC as part of their management culture continues to grow.
Accordingly, healthcare leaders and managers need to gain a good understanding of its implemen-
tation issues to ensure successful adoption as a strategic management tool in their
organizations.7,12
Recently a large urban non-teaching hospital in Canada serving a very diverse community, with
28,000 inpatients, more than 200,000 outpatients, and over 100,000 emergency visits and over
5000 births in 2010–2011, engaged in the development of a BSC for their Health Information
Management (HIM) department13 to align with all of the other departments that already had one.
This initiative to extend the BSC development to the HIM department was driven by the organi-
zation’s 3-year corporate strategy to pursue excellence in the delivery of integrated patient-centered
care while remaining focused on the continued building of an academic foundation. The develop-
ment of a BSC for the HIM department linked closely with the building of a strong academic
foundation focused on research, innovation, and scholarly activities and the expansion of education
to improve health outcomes. These strategic directions represent distinct dimensions in the corpo-
rate BSC. As such, the HIM BSC was derived with these larger corporate goals in mind, but also
included a third dimension to monitor and measure key enabling areas. Specifically, the enablers
were designed to monitor people, fiscal stewardship, health information technology and innova-
tion, communication, and partnerships.
Overall, the BSC development within the HIM department was precipitated by the organiza-
tion’s aim to provide all of their healthcare leaders and managers access to a fully electronic BSC
to equip them with the necessary metrics to support informed decision making.14,15
Methods
Sample
In all, 45 members of the HIM department comprised managers (9), analysts (7), and administra-
tive staff (29) who supported the BSC planning, development, and evaluation.
Instrument
Analytic hierarchy process questionnaire. To identify priority levels within the HIM BSC, the leader-
ship management team was asked to complete an analytic hierarchy process (AHP) questionnaire.
The AHP is a theory of measurement that is conducted through pairwise comparisons of alterna-
tives and is reliant on experts to identify priority scales that measure the intangibles inherent in
decisions in relative terms. The comparisons are made using a scale of absolute judgments that
represent the degree of dominance of one element over another with respect to a given attribute.16
It is beneficial in that it eases the effect of judgment biases, integrates the vast amount of informa-
tion in the BSC, and offers a comprehensive measure for performance assessment.7 The AHP is
useful in providing a structure as well as an algorithm to develop a comprehensive measure that is
useful for comparative purposes and is used to assess face and content validity within the final
indicators in the BSC. Construct validity could not be examined because of the absence of BSCs
4. Nippak et al. 123
for HIM departments within other hospitals across the country. Using the AHP as a tool, 10 ques-
tions were developed to address different aspects of the HIM BSC. This survey tool was distributed
online via Survey Monkey (Appendix 1).
Evaluation tool. The BSC evaluation tool consisted of 10 questions that were derived and adopted
from Barnardo and Jivanni’s17 formative evaluation conducted in 2009. The survey was divided
into four sections (Appendix 2). The first two questions collected demographic information from
the respondents. The next section asked questions regarding the HIM scorecard as a reporting tool.
Questions regarding the usefulness of the scorecard as a management tool were asked in section 3
of the tool. Finally, the last two questions required respondents to give their opinion on the second-
ary benefits of the HIM scorecard development activities.
Procedure
The construction of the HIM BSC involved three different phases: (1) planning, (2) development,
and (3) evaluation.
Planning phase. The creation of a HIM BSC began with a review of the hospital’s current strategic
plan and initiatives. To ensure alignment of the HIM BSC to the larger corporate BSC, the same
approach was used to create the HIM strategy map (Figure 1).
Some elements of the corporate strategy map (e.g. vision, mission, and strategic initiatives)
were modified and reworded to fit the function and strategic activities of the HIM department. An
exhaustive examination of the different functions and activities within the department was con-
ducted through review of relevant intranet web pages and interviews with managers and supervi-
sors from key HIM areas. This was done to ensure that the indicators selected within the BSC
would adequately and accurately measure the key functions and activities within each HIM area.
The HIM department provides a wide range of services, from processing and management of
patient health records to conducting strategic analysis of health information designed to the support
organizational planning.
The next step of the planning phase involved a review and update of the HIM department’s
organizational chart (Figure 2). Seven key areas were identified based on distinct business activi-
ties that were carried out throughout the department: (1) health records—coding and abstraction;
(2) health records—transcription; (3) health records—records processing; (4) decision support; (5)
patient flow and bed control; (6) registration; and (7) privacy, freedom of information, and release
of information.
Health Records is responsible for ensuring the accuracy and integrity of the patient health record
and also provides healthcare providers with access to health information critical in the delivery of
appropriate patient care. Three specialized areas report directly to the Health Records manager:
Coding and Abstraction (CA), Transcription, and Records Processing. These three departments
control administrative data and clinical data such as the patients’ medical history, reason for visit,
and diagnoses that are then captured and indexed into electronic records to be tracked. The Health
Records department uses the data captured to monitor clinical indicators such as infection rates,
cesarean section rates, and mortality rates. This allows them to review internal performance and
financial performance with respect to surgical volumes and cost per weighted case. The data from
the Health Records department are submitted to a number of registries that can be accessed and
then publicly reported by healthcare bodies such as the Canadian Institute for Health Information
(CIHI), Statistics Canada, the Institute for Clinical Evaluative Studies (ICES), and the Public
Health Agency of Canada (PHAC).
5. 124 Health Informatics Journal 22(2)
Decision Support is the analytic branch of the HIM department. As seen in Figure 2, the team is
made up of decision support analysts, program analysts, and strategic analysts. They offer leader-
ship and consultative services to various internal and external decision-makers by providing their
Figure 1. HIM Strategy Map 2012-2015.
HIM: Health Information Management.
6. Nippak et al. 125
expertise and knowledge of various data sets and databases. As such, they are responsible for the
validation of tools and reports that are used by both internal and external sources. They also respond
to ad hoc requests to support different management teams in the hospital, provide feedback and
analysis to the various leaders within the organization regarding their programs’ performance, and
monitor hospital performance while keeping leaders and managers in the hospital informed.
Based on information obtained from reviewing relevant intranet web pages, interviews with the
leadership management team, and observation of operational activities, process maps were devel-
oped for key areas to visualize the various business processes in the department. A Suppliers-
Inputs-Process-Outputs-Customers (SIPOC) chain model was used to construct a visual model of
the activities in each area (Figure 3). A SIPOC diagram is a simple high-level process mapping tool
often used within traditional Six Sigma projects.18 This tool has been proven to assist managers in
identifying critical stages in the business process needed to yield productive and valuable goods
and ideas.19 In order to develop key performance metrics for any HIM BSC, the literature indicates
that it is essential to create SIPOC diagrams for the relevant department areas. As such, SIPOC
diagrams were created for Coding and Abstraction (Figure 4), Transcription (Figure 5), Records
Processing (Figure 6), and Decision Support (Figure 7).
An information flow map was also constructed to identify key data sources for the performance
indicators (Figure 8). Five data sources were identified: (1) Finance, (2) Human Resources, (3)
Coding, (4) Cerner (the information technology management service provider), and (5) Data
Mart(s). The completion of this phase required 2 months.
Development phase. Good performance metrics have the characteristics of being relevant, under-
standable, timely, comparable, reliable, and cost effective,21 which formed the basis for the final
metrics selected. Using the SIPOC chain model developed in the initial phase, four major catego-
ries of performance indicators were identified: (1) input measures, (2) process measures, (3) output
measures, and (4) outcome measures. Within each major category, subcategories of indicators were
Figure 2. HIM department organizational chart.
HIM: Health Information Management.
7. 126 Health Informatics Journal 22(2)
created. An inventory of metrics was developed consisting of 52 potential indicators representing
all of the key areas in the HIM department. Performance indicators were narrowed down through
consultation with the director and members of the leadership management team. This selection and
refinement of indicators required 1.5 months to complete.
A total of 20 performance indicators made up the final HIM BSC design (Figure 9). Although
the individual performance indicators may vary from one institution to another, it is important to
consider including indicators from across the various categories of (1) input measures, (2) process
measures, (3) output measures, and (4) outcome measures. The final design and technical report
of the HIM BSC was presented to the director for review and approval. The technical report con-
tained key information about each indicator, which included a supporting definition, method of
calculation, any relevant reference, and additional explanation when required, target levels, the
data source used to derive the target levels, reporting responsibility, and status. To identify the
priority levels of the indicators within the HIM BSC, the leadership management team was asked
to complete theAHP questionnaire. This survey tool was distributed online for 2 weeks via Survey
Monkey. The distribution and analysis of the survey required 1 month to complete.
Figure 3. Supplier-input-process-output customer basic diagram.
Source: Office of Financial Management.20
Figure 4. SIPOC diagram—coding.
SIPOC: Suppliers-Inputs-Process-Outputs-Customers.
8. Nippak et al. 127
A meeting with 45 participants was organized to provide an opportunity for the HIM staff to
learn more about the corporate strategy and their role in helping achieve the organization’s goals
and initiatives. Members of the senior executive team were invited to talk about the hospital’s
strategic directions and the strategic plan for the next 3 years, which provided the opportunity to
demonstrate how the HIM BSC could support the larger hospital goals. The completion of the
entire development phase required 3 months.
Evaluation phase. Following the staff meeting, all participants were given a paper-based question-
naire to evaluate the utility of the BSC as a reporting tool and to determine the utility of the tool as
an effective management tool. The completion of this phase required 1 month.
Figure 5. Supplier-input-process-output customer diagram—transcription. Contractor performs a
random review a transcribed reports (5%). Internal review is also conducted.
Figure 6. Supplier-input-process-output customer diagram—records processing.
9. 128 Health Informatics Journal 22(2)
Results
The AHP questionnaire yielded the following results seen in Table 1. The table reflected prioritiza-
tion for performance, input, process, output, and outcome measures. The most important in terms
of priority was assigned to “outcome” when measuring performance workload. When measuring
the input measures, workload was the top priority, while the top priority for process measures was
“quality,” and “accuracy” was deemed to be the top priority for the output measures, while “com-
pliance of standards” was the top priority for the outcome measures.
Descriptive statistics were used to analyze the survey responses from the HIM staff following
the development of the BSC. There were 45 members of the HIM department who attended the
Figure 7. Supplier-input-process-output customer diagram—decision support.
Figure 8. Information flow map.
10. Nippak et al. 129
staff retreat with a final response rate of 93 percent (42 respondents). Respondents were catego-
rized based on their role and years of service. Of the 42 respondents, 14 percent (6) were members
of the leadership management team, 17 percent (7) were analysts, and 69 percent (29) were front-
line and administrative staff. Among the respondents, 2 percent (1) had been working at the hospi-
tal for less than a year, 12 percent (5) had been working between 1 and 5 years, 19 percent (8) had
been working between 6 and 10 years, and 67 percent (28) had been with the hospital for more than
10 years.
In order to assess the usefulness of the HIM BSC as a performance-reporting tool, respondents
were asked to provide their opinion on the number of indicators in the scorecard. In all, 83 percent
of the leadership management team indicated that the number of indicators in the scorecard was
“just right” compared to 43 percent of the analysts, and 83 percent of the frontline and administra-
tive staff, respectively. Overall, 76 percent of the respondents indicated that the number of indica-
tors in the HIM BSC was “just right.” Respondents were also asked whether the scorecard was easy
to understand. In all, 95 percent of all respondents indicated that the scorecard was “very easy” and
“somewhat easy to understand” and also agreed and strongly agreed to the statement: “The HIM
scorecard is a useful tool in reporting performance” (Table 2).
In order to assess the usefulness of the HIM BSC as a management tool, respondents were asked
to select the most suitable response to the following statement: “The HIM scorecard is a useful tool
in managing performance.” All of the leadership management team agreed and strongly agreed to
this statement compared to 71 percent of analysts and 72 percent of frontline staff. An overall 76
percent approval rating for the whole group was calculated. Respondents were also asked to
Figure 9. HIM balanced scorecard final design.
HIM: Health Information Management.
11. 130 Health Informatics Journal 22(2)
evaluate the level of alignment of the HIM scorecard indicators to the corporate strategy. Overall,
81 percent said that the indicators were completely and partially linked (Table 3).
The last section of the survey assessed staff views on the secondary benefits of the HIM BSC
development and implementation. Respondents were asked whether the scorecard development
activities made them feel engaged. Overall, 92 percent of the respondents indicated that they felt
engaged. Respondents were also asked whether the scorecard development activities enabled
members of the department to share common goals and objectives. A total of 92 percent of the
respondents answered “agree” and “strongly agree” to this question (Table 4).
Table 1. Analytic hierarchy process questionnaire results.
Normalized priority Idealized priority
1. Major performance indicators
Category Input 0.0550 0.0900
Process 0.2268 0.3700
Output 0.1119 0.1800
Outcome 0.6063 1.0000
2. Prioritized ratings for input measures
Category Staff 0.2335 0.3300
Workload 0.7014 1.0000
Technology 0.0651 0.0900
3. Prioritized ratings for process measures
Category Timeliness 0.1667 0.2500
Quality 0.6667 1.0000
Error rate 0.1667 0.2500
4. Prioritized ratings for output measures
Category Productivity 0.0000 0.0000
Accuracy 1.0000 1.0000
5. Prioritized ratings for outcome measures
Category Customer satisfaction 0.2475 0.3800
Compliance with standards 0.6447 1.0000
Staff engagement 0.1078 0.1700
HIM: Health Information Management.
Table 2. The HIM balanced scorecard as a performance-reporting tool.
Questions M (n = 6) A (n = 7) S (n = 29) All (N = 42)
Percent responding just right to the statement:
“The number of indicators in the scorecard is …”
83 43 83 76
Percent responding very easy or somewhat
easy to the question: “Is the scorecard easy to
understand?”
100 93 93 95
Percent agreeing or strongly agreeing to the
statement: “The HIM Scorecard is a useful tool
for reporting performance.”
100 86 97 95
M: leadership management team; A: analysts; S: frontline and administrative staff; HIM: Health Information Management.
12. Nippak et al. 131
Discussion and recommendations
The Canadian healthcare environment is rapidly changing. Thus, in order for healthcare organiza-
tions to optimize their successful delivery of services they must be responsive and actionable to
changewithintheenvironmentthroughtheutilizationofinnovativeandevidence-basedapproaches.
As a result, hospitals in Ontario are required to publicly report several performance indicators
within their BSC, which are largely focused directly on patient outcomes and quality of care, but
do vary between institutions.22 Thus, there exists some variability between hospitals, and, cur-
rently, HIM-specific indicators are not being captured within the BSC of any hospital because they
are not ministry reportable indicators. The inclusion of HIM indicators in a hospital BSC will allow
the leaders and managers in different levels of an organization to stay focused and well informed
about their departmental progress, as it will enable them to align their departmental operations with
the larger hospital BSC, and this will promote evidence-based management. The findings from the
evaluation of the HIM BSC indicated that the HIM managers, staff, and analysts felt that it was
both a useful performance-reporting tool and a useful management tool.
However, according to the literature, subsequent evaluation should be performed after a BSC
has been in use for a few years. As cited by Barnardo and Jivanni,17 according to Kaplan and
Norton, it takes 2–3 years for a BSC to fully integrate into the management culture. For this
reason, it is highly recommended that a formative evaluation be carried out to assess the effec-
tiveness of the tool in fulfilling the HIM departmental strategy a year or two after its implemen-
tation. As well, performance metrics should be reviewed and analyzed on a yearly basis to test
the relevance of each measure to the department’s strategy14,21 and the wider hospital’s corporate
strategy.
Table 3. The HIM balanced scorecard as a management tool.
Questions M (n = 6) A (n = 7) S (n = 29) All (N = 42)
Percent responding completely or partially to the
question: “Are the indicators in the scorecard
linked to NYGH’s strategic directions?”
100 86 76 81
Percent agreeing or strongly agreeing to the
statement: “The HIM Scorecard is a useful tool
in managing performance.”
100 71 72 76
M: leadership management team; A: analysts; S: frontline and administrative staff; HIM: Health Information Management;
NYGH: North York General Hospital.
Table 4. Secondary benefits of the HIM balanced scorecard.
Questions M (n = 6) A (n = 6) S (n = 26) All (N = 38)
Percent agreeing or strongly agreeing to the
statement: “The scorecard development
activities made me feel engaged.”
100 83 92 92
Percent agreeing or strongly agreeing to the
statement: “The scorecard development
activities enabled members of the department
to share common goals and objectives.”
100 83 92 92
M: leadership management team; A: analysts; S: frontline and administrative staff; HIM: Health Information Management.
13. 132 Health Informatics Journal 22(2)
Thus, the evaluation tools used in both the planning and development phases within this study
can be utilized within other healthcare organizations to assist with the mapping of key departmen-
tal processes and the selection of categories of performance metrics to support the measurement of
HIM activities. Capturing HIM-specific indicators will help gauge performance and promote pro-
cess improvement within an HIM department, however, the derived indicators within the current
study may not be entirely applicable to other organizations. The health information data within an
organization that is used to build an HIM BSC are strongly influenced by several hospital factors
that vary from one institution to another and will therefore influence the HIM performance indica-
tors within the BSC. These variations may result in different performance metrics for different
types of institutions. Therefore, the utility of the specific indicators as benchmarks may be limited
to hospitals that have similar characteristics to the hospital examined within the current study. In
general, BSC creation within an HIM department should focus on developing categories of perfor-
mance metrics, similar to those established within the current study, which are linked to input,
process, output, and outcome, but the specific metrics within each category may vary between
hospitals of different size and type.
Given the favorable response within the current study from all members of the HIM department,
active involvement of HIM staff in all BSC development activities is recommended. In conclusion,
the following recommendations are proposed for other HIM departments seeking to develop a BSC
based on the success of the current study:
•• Apply SIPOC mapping to effectively map departmental processes to aid in the development
of appropriate performance indicators.
•• Adopt the AHP to assist with the selection of final BSC indicators.
•• In addition to continuous process monitoring, conduct a formative evaluation after 2–3
years of BSC implementation.
•• Ensure active involvement of all members of the department in all phases of the develop-
ment, implementation, and continuous evaluation of a BSC.
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or
publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.
References
1. Kaplan RS and Norton DP. Using the balanced scorecard as a strategic management tool. Boston, MA:
Harvard Business Review, 1996.
2. Rohm H. Using the balanced scorecard to align your organization, http://balancedscorecard.org/
Portals/0/PDF/BalancedPerformance_Article1.pdf (accessed 23 October 2013).
3. Chan YL. Performance measurement and adoption of balanced scorecards: a survey of municipal gov-
ernments in the USA and Canada. Int J Pub Sect Manag 2004; 17: 204–222.
4. Kollberg B and Elg M. The practice of the balanced scorecard in health care. Int J Product Perform
Manag 2011; 60: 427–445.
5. Bigliardi B and Dormio A. A balanced scorecard approach for R&D: evidence from a case study.
Facilities 2011; 28: 278–289.
6. Soderberg M, Kalagnanam S, Sheehan NT, et al. When is balanced scorecard a balanced scorecard? Int
J Product Perform Manag 2011; 60: 688–708.
14. Nippak et al. 133
7. Chan YL. An analytic hierarchy framework for evaluating balanced scorecards of healthcare organiza-
tions. Can J Adm Sci 2006; 23: 85–104.
8. Kaplan RS and Norton DP. The balanced scorecard—measures that drive performance. Boston, MA:
Harvard Business School Publishing, 1992.
9. Chytas P, Glykas M and Valiris G. A proactive balanced scorecard. Int J Product Perform Manag 2010;
31: 460–468.
10. Kaplan RS and Norton DP. The strategy focused organization: how balanced scorecard companies
thrive. Boston, MA: Harvard Business Press, 2001.
11. Chan YL and Ho SK. Performance measurement and the use of balanced scorecard in Canadian hospi-
tals. Adv Manag Acc 2000; 9: 145–169.
12. McdonaldB.AreviewoftheuseofBSCinhealthcare,http://www.bmcdconsulting.com/index_htm_files/
Review%20of%20the%20Use%20of%20the%20Balanced%20Scorecard%20in%20Healthcare%20
BMcD.pdf (2012, accessed 23 October 2013).
13. North York General Hospital. We did. We are NYGH. 2012–2015 strategic plan, http://www.stcworks.
ca/north-york-general-hospital-strategic-plan/ (2012, accessed 24 September 2012).
14. Hodges B and Videtto D. Assessment and planning in health programs. 2nd ed. Sudbury, MA: Jones &
Bartlett Learning, 2012.
15. Muia M and Wong E. Linking strategy to operational execution through an enterprise business intel-
ligence system (PowerPoint Slides), http://www.ehealthconference.com/Presentations/e-Health%20
2012%20Presentations/O21.3.pdf (2012, accessed 24 September 2012).
16. Saaty T. Decision making with the analytic hierarchy process. Int J Serv Sci 2008; 1: 83–98.
17. Barnardo C and Jivanni A. Evaluating the Fraser Health Balanced Scorecard—a formative evaluation.
Healthc Manage Forum 2009; 22: 49–60.
18. Johnston M and Dougherty D. Developing SIPOC diagrams. ASQ Six Sigma Forum Mag 2012; 11:
14–18.
19. Nold H. Merging knowledge creation theory with six-sigma model for improving organizations: the
continuous loop model. Int J Manage 2011; 28: 479–477.
20. Office of Financial Management, State of Washington. Performance measure guide. Olympia, WA:
Washington State Office of Financial Management, 2009, p. 6.
21. Rossi P, Lipsey M and Freeman H. Evaluation: a systematic approach. 7th ed. Thousand Oaks, CA:
SAGE, 2007.
22. Weir E, d’Entremont N, Stalker S, et al. Applying the balanced scorecard to local public health perfor-
mance measurement: deliberations and decisions. BMC Publ Health 2009; 9: 127.
15. 134 Health Informatics Journal 22(2)
Appendix 1
Data collection tools
Analytic Hierarchy Process Survey Questionnaire
This survey was designed to help identify priority metrics in your departmental scorecard. In order
to achieve this goal, the analytic hierarchy process (AHP) was used to develop the following ques-
tions. The AHP is a structured decision-making framework that helps organize and analyze com-
plex decisions.
This survey has 10 questions and will take no more than 10 min to complete. Thank you for your
time.
1. Which is more important in evaluating the department’s performance?
Choices:
Input—measure of resources used by an activity or process
Process—describes aspects of the business process
Output—measure of units produced or service delivered
Outcome—measure of ultimate benefits associated with the product or service
Input OR Process
Input OR Output
Input OR Outcome
Process OR Output
Process OR Outcome
Output OR Outcome
2. Using a 9-point scale where
1—the performance metric is as important as the other;
3—the performance metric is weakly as important than the other;
5—the performance metric is strongly more important than the other;
7—the performance metric is demonstrably more important than the other;
9—the performance metric is absolutely more important than the other;
and 2, 4, 6, 8 are intermediate judgments,
please rate the relative importance of the metric you selected in the previous question versus
the one that you did not select.
16. Nippak et al. 135
3. Which is more important in evaluating the department’s input?
Choices:
Staff—percentage of staff hours used to carry-out the mandate of the area
Workload—volume of work to be completed
Technology—use of technology to enhance the business process
4. Using a 9-point scale where
1—the performance metric is as important as the other;
3—the performance metric is weakly as important than the other;
5—the performance metric is strongly more important than the other;
7—the performance metric is demonstrably more important than the other;
9—the performance metric is absolutely more important than the other;
and 2, 4, 6, 8 are intermediate judgments,
please rate the relative importance of the metric you selected in the previous question versus the
one that you did not select.
Input/Process 1 2 3 4 5 6 7 8 9
Input/Output 1 2 3 4 5 6 7 8 9
Input/Outcome 1 2 3 4 5 6 7 8 9
Process/Output 1 2 3 4 5 6 7 8 9
Process/Outcome 1 2 3 4 5 6 7 8 9
Output/Outcome 1 2 3 4 5 6 7 8 9
Staff OR Workload
Staff OR Technology
Workload OR Technology
Staff/Workload 1 2 3 4 5 6 7 8 9
Staff/Technology 1 2 3 4 5 6 7 8 9
Workload/Technology 1 2 3 4 5 6 7 8 9
5. Which is more important in evaluating the department’s process?
Choices:
Timeliness—time required to complete the business process/cycle
Quality—percentage of work that does not require re-work
Error Rate—percentage of work flagged with errors
17. 136 Health Informatics Journal 22(2)
6. Using a 9-point scale where:
1—the performance metric is as important as the other;
3—the performance metric is weakly as important than the other;
5—the performance metric is strongly more important than the other;
7—the performance metric is demonstrably more important than the other;
9—the performance metric is absolutely more important than the other;
and 2, 4, 6, 8 are intermediate judgments,
please rate the relative importance of the metric you selected in the previous question versus the
one that you did not select.
Timeliness OR Quality
Timeliness OR Error Rate
Quality OR Error Rate
Timeliness/Quality 1 2 3 4 5 6 7 8 9
Timeliness/Error Rate 1 2 3 4 5 6 7 8 9
Quality/Error Rate 1 2 3 4 5 6 7 8 9
Productivity OR Accuracy
7. Which is more important in evaluating the department’s output?
Choices:
Productivity—average number of work completed per given time
Accuracy—degree of specificity/accuracy of information captured
8. Using a 9-point scale where
1—the performance metric is as important as the other;
3—the performance metric is weakly as important than the other;
5—the performance metric is strongly more important than the other;
7—the performance metric is demonstrably more important than the other;
9—the performance metric is absolutely more important than the other;
and 2, 4, 6, 8 are intermediate judgments,
please rate the relative importance of the metric you selected in the previous question versus the
one that you did not select.
Productivity/Accuracy 1 2 3 4 5 6 7 8 9
18. Nippak et al. 137
9. Which is more important in evaluating the department’s output?
Choices:
Customer satisfaction—customer rating of service delivery
Compliance with standards—degree of compliance with industry standards
Staff engagement—overall rating of staff engagement
10. Using a 9-point scale where:
1the performance metric is as important as the other;
3—the performance metric is weakly as important than the other;
5—the performance metric is strongly more important than the other;
7—the performance metric is demonstrably more important than the other;
9—the performance metric is absolutely more important than the other;
and 2, 4, 6, 8 are intermediate judgments,
please rate the relative importance of the metric you selected in the previous question versus the
one that you did not select.
Customer Satisfaction OR Compliance with Standards
Customer Satisfaction OR Staff Engagement
Compliance with Standards OR Staff Engagement
Customer Satisfaction/Compliance 1 2 3 4 5 6 7 8 9
Customer Satisfaction/Staff Engagement 1 2 3 4 5 6 7 8 9
Compliance/Staff Engagement 1 2 3 4 5 6 7 8 9
19. 138 Health Informatics Journal 22(2)
Appendix 2
Health Information Management balanced scorecard Evaluation Survey
Demographics
1. Please select one of the following that describes your role:
□ Director/Manager/Coordinator/Supervisor
□ Analyst
□ Frontline Staff/Administration
□ Others (please specify): ________________________________
2. How many years have you been with North York General Hospital?
□ Less than a year
□ 1 to 5 years
□ 6 to 10 years
□ More than 10 years
The Health Information Management (HIM) scorecard as a reporting tool
3. The number of indicators in the scorecard is:
□ Just right
□ Too many indicators
□ Too few indicators
□ Not Sure
□ Not Applicable
4. Is the scorecard easy to understand?
□ Very easy to understand
□ Somewhat easy to understand
□ Not easy to use
□ Too easy—losing usefulness
□ Not applicable
5. Please select the most suitable response to the following statement:
“The HIM Scorecard is a useful tool for reporting performance.”
□ Strongly agree
□ Agree
□ Neutral
□ Disagree
□ Strongly disagree
20. Nippak et al. 139
The HIM scorecard as management tool
6. Are the indicators in the HIM scorecard linked to North York General Hospital (NYGH)’s
strategic directions?
□ Not linked
□ Partially linked
□ Completely linked
□ Not sure
□ Linking is not needed
□ Not aware of NYGH strategic directions
□ Not applicable
7. How will the HIM scorecard information be used? Please select all that apply.
□ Compare performance with peers, benchmark, and MOH expectations
□ Compare with own past performance
□ Identify performance improvement strategies
□ Assign responsibility for performance improvement
□ Show the effects of performance improvement strategies
□ Other (please specify):
____________________________________________________________
8. Please select the most suitable response to the following statement:
“The HIM Scorecard is a useful tool in managing performance.”
□ Strongly agree
□ Agree
□ Neutral
□ Disagree
□ Strongly disagree
Secondary benefits of the HIM scorecard development and implementation
9. The scorecard development activities made me feel engaged.
□ Strongly agree
□ Agree
□ Neutral
□ Disagree
□ Strongly disagree
10. The scorecard development activities enabled members of the department to share common
goals and objectives.
□ Strongly agree
□ Agree
□ Neutral
□ Disagree
□ Strongly Disagree
