This presentation was put together on the special topic of measurement when a group from Sweden was visiting the Dept of Veterans Affairs National Center for Patient Safety to learn about patient safety improvement programs underway there. I remember some of the people listening resisting my main point that so far there was no good way to measure PS outcomes, but some good ways to measure important outcomes that are potential precursors to patient safety problems (like not getting X-rays verified in a timely way).

1. 1
Presentation for the
Stockholm County Council
Noel Eldridge, MS
Department of Veterans Affairs
National Center for Patient Safety
(202) 461-6517 & noel.eldridge@va.gov
11/1/07

2. 2
VA’s Current Patient Safety
Performance Measure
Radiology Reports
verified within 2 days
• Starting point in 2005
was 67%
• Most recent quarter
was 90%
• Reports in first 3 Qtrs of
FY 2007: 5,255,967
• The difference between
67% and 90% is over
1,600,000 reports per
year, or ~4,400 per day
everyday.

3. 3
Why this is important…
• RCA counts for 3 years
shown
• Delay in Treatment/
Diagnosis/Surgery is tied
with Patient Falls for most
reported coded category
of Root Cause Analysis
submitted by VAMCs
Delay in Treatment/Diagnosis/Surgery

4. 4
Using Administrative Data to
Measure Patient Safety
• This would be great if it worked (in particular
the AHRQ Patient Safety Indicators [PSIs]).
• At least three major problems:
1. False Negatives (a new version of the classic
under-reporting problem plaguing patient safety)
2. False Positives (the new problem based on “present
on admission” and dependence on coding)
3. How to analyze data:
– designed for another purpose,
– characterized by rare events, and
– is often “non-Gaussian”, i.e., not “normally”
distributed

5. 5
Under-reporting?
adverse events, complications, or illnesses?
CDC’s Official Number of Deaths
for Leading Causes of Death
http://webapp.cdc.gov/sasweb/ncipc/leadcaus10.html
• Heart disease: 652,486
• Cancer: 553,888
• Stroke (cerebrovascular
diseases): 150,074
• Chronic lower respiratory
diseases: 121,987
• Accidents (unintentional injuries):
112,012
• Diabetes: 73,138
• Alzheimer's disease: 65,965
• Influenza/Pneumonia: 59,664
• Nephritis, nephrotic syndrome,
and nephrosis: 42,480
• Septicemia: 33,373
CDC Analysis published in
Public Health Reports,
March - April 2007, pp 160-166
The estimated deaths associated
with healthcare-associated infections
[for 2002] in U.S. hospitals were 98,987:
• 35,967 were for pneumonia,
• 30,665 for bloodstream infections,
• 13,088 for urinary tract infections,
• 8,205 for surgical site infections, and
• 11,062 for infections of other sites.
These are from completely separate analyses…

6. 6
False Negatives?
• 2002 VA data, from
Rosen, et al, 2006, cases
found:
– Complications of
Anesthesia = 55
– “Infection resulting from
Medical Care” (highly
restricted definition) =
816
– Postoperative Sepsis =
106
• VA performs over
350,000 operations per
year… compare to PSI
denominators

7. 7
False Positives?
- Decubitus Ulcer = 3316
- AHRQ analysis of data from
California and NY indicates
that >80% were present on
admission
- Post-op Hip Fracture = 29
- AHRQ data indicates >70%
present on admission
- A false positive and a false
negative problem? (29 total?
Leaves 10 in-hospital?)
- Mayo Clinic study
indicated that overall, only
63% occurred in
hospital…

8. 8
Recent Publication (not from AHRQ -- AHRQ paper in review)
Impact of Diagnosis-Timing Indicators on Measures of Safety, Comorbidity, and Case Mix Groupings From
Administrative Data Sources.
Medical Care. 45(8):781-788, August 2007.
Naessens, James M. ScD, MPH *; Campbell, Claudia R. PhD +; Berg, Bjorn BA *; Williams, Arthur R. PhD *;
Culbertson, Richard PhD +
Abstract:
Context: Many attempts to identify hospital complications rely on secondary diagnoses from billing data. To be
meaningful, diagnosis codes must distinguish between diagnoses after admission and those existing before
admission.
Objective: To assess the influence of diagnoses at admission on patient safety, comorbidity, severity measures, and
case mix groupings for Medicare reimbursement.
Design: Cross-sectional association of various diagnosis-based clinical and performance measures with and without
diagnosis present on admission.
Setting: Hospital discharges from Mayo Clinic Rochester hospitals in 2005 (N = 60,599).
Patients: All hospital inpatients including surgical, medical, pediatric, maternity, psychiatric, and rehabilitation patients.
About 33% of patients traveled more than 120 miles for care.
Main Outcome Measures: Hospital patient safety indicators, comorbidity, severity, and case mix measures with and
without diagnoses present at admission.
Results: Over 90% of all diagnoses were present at admission whereas 27.1% of all inpatients had a secondary
diagnosis coded in-hospital. About one-third of discharges with a safety indicator were flagged because of a
diagnosis already present at admission, more likely among referral patients. In contrast, 87% of postoperative
hemorrhage, 22% of postoperative hip fractures, and 54% of foreign bodies left in wounds were coded as in-hospital
conditions. Severity changes during hospitalization were observed in less than 8% of discharges. Slightly over 3% of
discharges were assigned to higher weight diagnosis-related groups based on an in-hospital complication.
Conclusions: In general, many patient safety indicators do not reliably identify adverse hospital events,
especially when applied to academic referral centers. Except as noted, conditions recorded after
admission have minimal impact on comorbidity and severity measures or on Medicare
reimbursement.
(C) 2007 Lippincott Williams & Wilkins, Inc.

9. 9
n= 1800+
Mean = 1.0
Std. Dev. = 1.8
Median = 0
Non-normally Distributed “Rare” Events
3+ years of data for nursing home sections of VA Medical
Centers – statistical analysis based on a normal distribution
is wrong and/or useless here: O/E ratios, p values, etc.

10. 10
Triangle Shirtwaist Factory
Fire of 1911
• March 16, 1911 A report on fire traps is published. The report argues
that many New York City buildings lack “even the most indispensable
precautions necessary.”
• March 25, 1911 Shortly before quitting time, a fire breaks out on the
eighth floor of the building housing the Triangle Shirtwaist factory. The
fire kills 146 victims.
• April 2, 1911 A meeting is held to discuss concerns over lack of safe
working conditions in New York City's factories. Resolutions are passed
demanding new legislation.

11. 11
What a Triangle Shirtwaist Factory
Fire Every Day Would be in U.S.
Healthcare
• 146 x 365 = 53,290 deaths
• 53,290 / 6,007 = 8.9 per year per hospital
• 365 / 8.9 = 41 days between deaths
or
• 53,290 / 36,600,000 = 0.15%
(1 in 675 admissions)
• Not so hard to believe… but would be 2.1% of all
deaths in US (53,290 / 2,443,387)
• IOM said 44,000 to 98,000 in 1999. Estimates
since have ranged from about 15,000 to over
200,000.