In FDA regulated industries, audits are high-stakes, fact-finding exercises required to verify compliance to regulations and an organization’s internal procedures. Although exploratory testing has emerged as a powerful test approach within regulated industries, an audit is the impact point where exploratory testing and regulatory worlds collide. Griffin Jones describes a heuristic model—Congruence, Honesty, Competence, Appropriate Process Model, Willingness, Control, and Evidence—his team used to survive an audit. You can use this model to prepare for an audit or to baseline your current practices for an improvement program. Griffin highlights the common misconceptions and traps to avoid with exploratory testing in your regulated industry. Avoid mutual misunderstandings that can trigger episodes of incongruous behavior and an unsuccessful audit. Learn how to maintain your composure during a stressful audit and leave with valuable heuristics to help you organize and present your exploratory testing results with confidence.
WSO2Con2024 - WSO2's IAM Vision: Identity-Led Digital Transformation
Surviving an fda audit griffin jones - nov 2011 - bspe
1. The Heuristics for Exploratory Testing
November 2011 1Griffin Jones – Congruent Compliance LLC ® 2011
2. Preliminaries
Who is in the room?
My goal:
Stimulate your interest to study the subject more
Leave with a heuristic to help you organize and present
with confidence your ET results to regulatory auditors
Have a conversation and try to meet your needs
Quick Preview
The context
The heuristic and how to apply it
Some of the traps about ET in a regulated industry
2Griffin Jones – Congruent Compliance LLC ® 2011November 2011
3. Assumptions and Terms
More reference information here than I will present
Follow the for the key points
Much of this can be adapted to other contexts
i.e., not “FDA regulated, Exploratory Testing”
“Schools of Testing” by Bret Pettichord
Analytic , Standard, Quality, Context-Driven, Agile
Exploratory Testing
Simultaneous learning, test design and test execution
Agile Testing
Story completion, test automation: Test Driven Dev., etc.
3Griffin Jones – Congruent Compliance LLC ® 2011November 2011
4. Terms
Congruence
Being balanced between inner feelings & outer actions
Smells
Symptom that possibly indicates a deeper problem
5 Whys
Questions-asking method to investigate root causes
“Mary had a little lamb” heuristic
Emphasize each of the individual words in a statement
Checking: confirming existing beliefs; versus:
Testing - finding new information (Michael Bolton)
4Griffin Jones – Congruent Compliance LLC ® 2011November 2011
5. The Problem
Let’s assume that you are FDA regulated and trying to
do compliant context-driven or Agile, Exploratory
Testing
You likely have these concerns about passing an audit:
Evidence is not sufficient
Documentation is not sufficient
Process control is not sufficient
Can’t clearly explain what you do and why
Auditors value different things than you, and speak a
different language
5Griffin Jones – Congruent Compliance LLC ® 2011November 2011
6. Fast Takeaway
The regulator is not your business partner
The regulator has police powers
Pick your battles – Sometimes, “Let the Wookie win”
“Render unto Caesar, that which is Caesar’s …”
Auditors are likely of the “Quality” (gatekeepers) or
“Routine” (traceability matrix) testing school model
You are a different testing school. Deal with it.
Auditors think “testing” is “demonstration and
checking”
Don’t try and convert them. Deal with it.
6Griffin Jones – Congruent Compliance LLC ® 2011November 2011
7. Spoiler
The regulations are not the problem
How you are coping with the regulations is the problem
Give the Auditors what they want:
Clear traceable requirements and description of risks
Description and demonstration of control
Clear objective evidence
The ability to understand their concerns, speak their
language, and explain how you are compliant
Abundant, quality evidence mitigates your other
problems
7Griffin Jones – Congruent Compliance LLC ® 2011November 2011
8. Not going to talk about…
The Fear, Uncertainly, and Doubt swirling in the field
Vendor/Experts: “You should be scared, but I have…”
Silver Bullets and Big Magic
“… so trust me and just buy my wares. By the way, ..”
Persistent Myths
“… IMO the regulators “frown on” ET (… I don’t sell it).”
The “Typical” Regulatory Affairs Presentation
8Griffin Jones – Congruent Compliance LLC ® 2011November 2011
9. Regulatory Overview
Regulations
For the public good - because people died
Regulators
FDA regulates >25% of the Gross Domestic Product
Regulatory Auditors
Police Powers
Industry Auditors
Assessors and valued advisors to management
Audits
9
Details
Griffin Jones – Congruent Compliance LLC ® 2011November 2011
10. Audit Survival Heuristics
CHCMWCE
“Chocolate Mousse”
Congruent
Honest
Competent
Model (Appropriate)
Willing
Control
Evidence
15
Model
Competent
Honest
Evidence
Control
Willing
Congruent
Griffin Jones – Congruent Compliance LLC ® 2011November 2011
11. Let’s take a journey …
16
Practice
Congruent
Theory
Less Stressful
Audits
Griffin Jones – Congruent Compliance LLC ® 2011November 2011
12. The Congruence Triad
Congruence is when you are balanced between inner
feelings and outer actions
The Congruence Triad
Self, Other, Context
Being congruent is a process
A way of communicating with yourself and others
Incongruence is when part of the triad is missing
Placating, Blaming, Super-rational, or Irrelevant?
What is missing and fill it in:
Self, Others, Context
17
Other
Context
Self
Details
Griffin Jones – Congruent Compliance LLC ® 2011November 2011
13. Congruence is like a Sailboat
Because:
It is a vessel or container, like a basket
It requires preparation and maintenance
You don’t “drive” it, and requires skills of crew members
Subject to weather
Is vulnerable to sinking
19Griffin Jones – Congruent Compliance LLC ® 2011November 2011
Tools
14. The Theory Mountains …
Dishonest
Incompetent
Inadequate
21
Honest
Competent
Appropriate Model
Self-Incriminating
Experts and Heroes
Over-Constrained
Griffin Jones – Congruent Compliance LLC ® 2011November 2011
15. Honest
Integrity, Truthful, Trust, Sincerity in:
You and your organization
Words, actions, and documents
Smells
Dishonest
Self-incrimination
Don’t create even the appearance of a problem
Tests
How do you and the organization react to criticism?
Are you a learning organization? (5 Why)
22Griffin Jones – Congruent Compliance LLC ® 2011November 2011
16. Competent
Are you and your organization:
Capable, credible, understands context, speaks the
language; trained in the industry, technology, and
regulatory obligations
Smells
Incompetent
Experts and heroes
Tests
Do you believe you are capable of doing good work?
(5 Why)
23Griffin Jones – Congruent Compliance LLC ® 2011November 2011
17. Appropriate Model
Is the process model:
Complete, reasonable, practical, logical, explainable
Smells
Inadequate model
Over-constrained model
Test:
What problem is this model solving? How will it Fail?
What is required in this model? Missing?
Do you believe this model is sufficient? (5 Why)
24Griffin Jones – Congruent Compliance LLC ® 2011November 2011
18. The Practice Mountains …
Unwilling
Out-of-Control
No Evidence
25
Excessive or Wasteful
Micro-Management
Obsessive-Compulsive
Willing
Under Control
Evidence
Griffin Jones – Congruent Compliance LLC ® 2011November 2011
19. Willing
Motivated, focused, prioritized, committed, resourced,
staffed, supported, given attention, nurtured
Smells
Unwilling
Excessive or Wasteful
Test
Do people care? (5 Why)
Is there sufficient resources for the work and
expectations? (5 Why)
26Griffin Jones – Congruent Compliance LLC ® 2011November 2011
20. Under Control
Explain what you are doing and why. Are you living it?
Coherently explain your:
configuration control and authorization
traceability and accountable
organization, preparation, planning, independent review,
prevention, correction, checking and testing
Smells
Out-of control
Micro-managed
Tests
Is the type and level of controls appropriate? (5 Why)
27Griffin Jones – Congruent Compliance LLC ® 2011November 2011
21. Evidence
Auditable evidence:
Clear, objective, retrieval, human readable, attributable,
contemporary evidence that a third party can review or
reconstruct (with minimal outside help); and quickly
reach the same results and conclusions.
Smells
No-evidence
Obsessive-compulsive evidence
Tests
Explain why the specific evidence meets the criteria.
(5 Why)
28Griffin Jones – Congruent Compliance LLC ® 2011November 2011
22. How do you apply this?
Application is as simple as:
29
Remembering
to ask the questions.
Follow the energy
of the answers.
Fix the base, first.
Griffin Jones – Congruent Compliance LLC ® 2011November 2011
23. During an Audit
Choosing a regulatory posture
Manageable issues (within reason)
Evidence
Controls
Willingness (resources and priority)
Unmanageable issues
Broken process model
Lack of competence
Broken trust
Incongruence
30Griffin Jones – Congruent Compliance LLC ® 2011November 2011
24. More Fast Takeaways
The FDA is open to agile processes and realizes that
the current approach to software validation is not
working
At the same time, companies are more concerned
about:
the business risk that the FDA would not accept the
agile process,
than the product or project risk that is associated with
waterfall type development
Find the middle option for your context
31Griffin Jones – Congruent Compliance LLC ® 2011November 2011
25. Natural Evidence
Periodically , take the observer point-of-view and ask:
Is what I see and hear, about the theory and practice of
what we do:
acceptable from both a product qualification and
regulatory compliance point of view?
If yes, what is the most natural, efficient, and strongest
evidence we could collect?
Why not a video/audio recordings w/ paper summary?
Is it being collected? If no, why not? (5 Why)
organizational problem?
32Griffin Jones – Congruent Compliance LLC ® 2011November 2011
27. Smells that lead to …
Stop Shaking the Snow Globe
Hyper-change alongside brittle/heavy formal processes
The “Best Practice” Cargo Cult
We don’t really understand the details of what we do,
why we do it, or how what we do works. But have faith.
Testing Death Spiral
Regulator does not care about testing and management
might only care about regulatory compliance. Spiral.
The Titanic
The gigantic engineered process is perfect – people are
the source of problems, not solutions
34Griffin Jones – Congruent Compliance LLC ® 2011November 2011
28. Organizational Disasters
Pathetic Compliance
Following a regulatory compliant procedure in a way
that does not solve the testing problem for which it was
designed.
Utopian Shelf-ware Procedures
No one reads them. They are not reality.
Close Enough
I don’t have to do it exactly. I know better. No one will
notice or care.
Read My Mind
Because that is the only place where the evidence is.
35Griffin Jones – Congruent Compliance LLC ® 2011November 2011
29. Is the Auditor on Tilt?
36
Maybe it is something we said or did, or are doing?
History
That you are unaware of, and it might be complicated
Notches on the gun
May be making a name for themselves
Making an example of you
May be constructing an example to deter others
Griffin Jones – Congruent Compliance LLC ® 2011November 2011
30. Classic Agile Traps
Mixing informal and formal processes
Start informal - clearly switch to formal when ready
Emphasizing change; light documents = poke the bear
Stokes anxiety: control, process model, and competence
Mistaking team conversation and understanding
For objective documented evidence
Speaking “Crazy Agile Moon Language”
Give the auditor what they want, in their language
Shows empathy and industry competence
37Griffin Jones – Congruent Compliance LLC ® 2011November 2011
Pass
Fail
31. Classic ET Traps
Implementation details identified as requirements
Tighten and simplify your requirements
Documentation lacks detail to support traceability
Require less mind reading
Control is vague or assumed
Summarize and document what control is for you
38Griffin Jones – Congruent Compliance LLC ® 2011November 2011
32. The BIG Trap
Weak Evidence
“Clear, objective, retrieval, human readable,
attributable, contemporary evidence that a third party
can review or reconstruct (with minimal outside help);
and quickly reach the same results and conclusions.”
Check it via “Mary had a little lamb”
Collect it naturally
Weak evidence is likely a symptom of other deeper
issues
Abundant, quality evidence mitigates your other
problems
39Griffin Jones – Congruent Compliance LLC ® 2011November 2011
33. Audits can be Useful
Candor can result in free consulting and insight
Should you take the risk?
Provides motivation – management cares
Provides actionable data
The jiggle that is needed by the organization
A counter-measure to low expectations & poor practices
40
If you can’t be a good example,
you are going to be a stern warning.
Griffin Jones – Congruent Compliance LLC ® 2011November 2011
34. Recap of the Spoiler
The regulations are not the problem.
How you are coping with the regulations is the problem.
Give the Auditors what they want:
Clear traceable requirements and description of risks
Description and demonstration of control
Clear objective evidence
The ability to understand their concerns, speak their
language, and explain how you are compliant
Abundant and quality evidence mitigates your other
problems.
41Griffin Jones – Congruent Compliance LLC ® 2011November 2011
35. The Big Take Away
Understand your regulatory context
Work on your congruence
Work each level of the model, ask the questions
Document how you are under control
Improve your evidence, collect it naturally
Avoid the smells, disasters, and traps
Summarize your regulatory story, practice explaining it
Apply what you learn during the audit
42
1
2
3
Griffin Jones – Congruent Compliance LLC ® 2011November 2011
37. Further Study - A
FDA presentations and resources:
Webinar with FDA's John Murray on Software Validation
in the Field of Medical Devices
Presentation: Preparing for an FDA Medical Device
Sponsor Inspection
Quality System Inspection Technique – Inspection
Guide
General Principles of Software Validation; Final
Guidance for Industry and FDA Staff
44Griffin Jones – Congruent Compliance LLC ® 2011November 2011
38. Further Study - B
Regulatory Compliance
“The Art of Compliance: Turning Compliance into
Sustainable Business Advantage” by Robert Rhoades of
Quintiles
FDA inspections:
“How to Host an FDA Inspection” by SGS – Life Science
Services
“Preparation for FDA Inspection” by
NEMA/ADVAMED/PHILIPS
“FDA Sponsor Inspections: How to Prepare and Survive”
by Medtronic, Inc
45Griffin Jones – Congruent Compliance LLC ® 2011November 2011
39. Further Study - C
Audits
“The ASQ Auditing Handbook” by J. P. Russell
Congruence
“Beyond Blaming” by Jean McLendon and Gerald M.
Weinberg
“The Satir Model: Family Therapy and Beyond” by Virginia M.
Satir
“More Secrets of Consulting: The Consultant's Tool Kit” by
Gerald M. Weinberg
Testers and Auditors
“Testers are like auditors” by James Christie
Evidence
“21 CFR Part 11 Electronic Records …” by the FDA
46Griffin Jones – Congruent Compliance LLC ® 2011November 2011
40. Further Study - D
Agile and the FDA
Business Risk (from the FDA) versus Product Risk
http://blogs.construx.com/forums/t/432.aspx
“What is Exploratory Testing? And How it Differs from
Scripted Testing” by James Bach
“Coping With Complexity: Lessons From a Medical Device
Project” by Yaron Kottler
“Introduction into IEC 62304 Software life cycle for medical
devices” by Christoph Gerber
http://www.spiq.com/abs/JF200809IEC62304%20SPIQ%20
Rev004.pdf
“Who says ET is good for Medical Devices? The FDA!” by
James Bach
http://www.satisfice.com/blog/archives/602
47Griffin Jones – Congruent Compliance LLC ® 2011November 2011
41. Further Study - E
Agile and the FDA
http://rdn-consulting.com/blog/2007/07/25/update-
agile-development-in-a-fda-regulated-setting/
http://www.agilejournal.com/articles/columns/column-
articles/3463-four-reasons-medical-device-companies-
need-agile-development
http://rdn-consulting.com/blog/wp-
content/uploads/2007/07/060703ResMed.pdf
http://scalingsoftwareagility.wordpress.com/2010/11/23/
an-iterative-and-incremental-process-model-for-agile-
development-in-regulated-environments/
http://scalingsoftwareagility.wordpress.com/category/hi
gh-assurance-and-regulated-environments/
48Griffin Jones – Congruent Compliance LLC ® 2011November 2011