In your routine laboratory works, do you have to issue a statement of conformity after testing a product sample, such as stating a Pass or fail, a compliance or non-compliance? What is your decision rule as required by this ISO standard? What is the risk to make a wrong decision in rejecting the test result based on the product specification limit when it is actually in conformance? Are you able to control such a risk in order to make an informed decision?
If you have all these questions in mind, I have got the answers for you.
3. Introduction
• Laboratory analysis is part of conformity assessment
• Conforming with the product specification stated in a sales contract
• Attesting the test result in compliance with a regulatory limit or tolerance level
• General statement of conformity includes:
• Conformance vs Non-conformance
• Compliance vs Non-compliance
• Acceptance vs Rejection
• Pass vs Fail
• Positive vs negative
• Presence vs absence
• A decision based on the tests and measurements must be made before issuing such statement. A set
of decision rules is to be established.
3
4. Introduction
• There is always a risk to make a wrong decision
• Tests done on a laboratory sample; how representative is the
sample?
• Test results are always associated with measurement
uncertainty
• Decision may be “biased” due to someone’s possible vested
interest
• Are we able to control such risk in order to make an
informed decision?
4
5. Definitions of “Decision Rule”
• ISO/IEC 17025:2017 clause 3.7: “Rule that describes how
measurement uncertainty is accounted for when stating conformity
with a specified requirement”
• ISO Guide 98-4, JCGM 106: “Documented rule that describes how
measurement uncertainty will be accounted for with regard to
accepting or rejecting an item, given a specified requirement and the
result of a measurement”
• Eurachem Guide on Compliance Assessment: “A documented rule
that describes how measurement uncertainty will be allocated with
regard to accepting or rejecting a product according to its
specification and the result of a measurement”
5
6. Therefore, decision rule is a rule taking measurement
uncertainty into account when stating conformity
with a specification value or compliance with a
regulatory limit.
When making a claim of non-conformity, the risk should be as
low as possible for false rejection, i.e., we should have high
confidence to correctly reject the test result.
6
8. Relevant clauses of ISO/IEC 17025:2017
• Clause 6.2.6 requires that the laboratory shall authorize
personnel to “analyse results, including statements of
conformity or opinions and interpretations”. That implies only
authorized signatories are allowed to handle decision making
in this case.
• The laboratory is now required to record the decision rule
adopted taking into account the risk; such a rule will have on
reporting false positive or negative results. (Ref: clause 7.1.3)
8
9. Relevant clauses 7.1.3 of ISO/IEC 17025:2017
• 7.1 Review of requests, tenders and contracts
• 7.1.3 When a customer requests a statement of conformity
to a specification or standard for the test or calibration (e.g.
pass/fail, in-tolerance/out-of-tolerance) the specification or
standard, and the decision rule shall be clearly defined. Unless
inherent in the requested specification or standard, the
decision rule selected shall be communicated to, and agreed
with, the customer.
9
10. Relevant clauses 7.8.6 of ISO/IEC 17025:2017
• 7.8.6 : Reporting statements of conformity
• 7.8.6.1 When a statement of conformity to a specification
or standard is provided, the laboratory shall document the
decision rule employed, taking into account the level of
risk (such as false accept and false reject and statistical
assumptions) associated with the decision rule employed,
and apply the decision rule.
• NOTE: Where the decision rule is prescribed by the
customer, regulations or normative documents, a further
consideration of the level of risk is not necessary.
10
11. Relevant clauses 7.8.6 of ISO/IEC 17025:2017
• 7.8.6 Reporting statements of conformity (contd.)
• 7.8.6.2 The laboratory shall report on the statement of conformity, such
that the statement clearly identifies:
• a) to which results the statement of conformity applies;
• b) which specifications, standards or parts thereof are met or not met;
• c) the decision rule applied (unless it is inherent in the requested
specification or standard).
The laboratory shall report on the statement of conformity, such that the
statement clearly identifies:
• To which results the statement of conformity applies;
• Which specifications, standards or parts thereof are met or not;
• The decision rule applied
11
12. To make a decision rule, we must have….
• A measurand (analyte) clearly specified (i.e. specification of the
object)
• A test result (normally assuming normal distribution of test results)
• Its associated measurement uncertainty
• A specification or regulation giving upper and/or lower limits
• A decision rule
This rule can decide to take or not to take measurement uncertainty
into account, but it includes the risk of making a wrong decision,
of which the involved parties are willing to take.
12
13. Aims of making a decision rule
• One should be clear on what type of risk he is taking
when making a proper decision rule:
• the supplier’s (laboratory’s) risk (Type I error, ) or
• the consumer’s (customer’s) risk (Type II error, )
• From the laboratory point of view, we should be looking
at Type I () False Positive error, as we want to minimize
and control our risk in issuing a statement of conformity
that might be proven wrong later.
• We usually set the error or risk that we can afford to take,
e.g. = 0.05 (or 5%), bearing in mind that our test results
have associated measurement uncertainties.
13
Important assumption: The
uncertainty of measurement
is represented by a normal
(Gaussian) probability
distribution function, which is
consistent with the typical
measurement results (being
assumed the applicability of
the Central Limit Theorem)
14. What is Type I () error?
• A type I error is caused by wrongly rejecting a true (“Pass”) situation
(statistically speaking, a null hypothesis Ho)
• This happens when the test result is found to be close to the specification
value.
• In a hypothesis (significance) testing, we write:
• Null hypothesis Ho : test result = specification value (not literally but with probability)
• Alternate hypothesis H1 : test result > or < specification value (depending on the
upper or lower limits of specification)
• Generally, we want to control our error (risk) when making a statement to
reject Ho, i.e., saying the test result fails to meet with the given specification
limit
• We normally fix such error at = 0.05, or 5% error (or 95% confidence)
14
15. Today’s practice on decision rules
• In current practice, we have been making a very simple decision
rule, often based on direct comparison of measurement value
with the specification or regulatory limits.
• The reason is either that
• these limits are deemed to have assumed including the measurement
uncertainty, which is not normally true,
• or, it has been assumed that the measurement value has zero
uncertainty!
• But, due to the presence of uncertainty in all measurements,
we are actually taking more than 50% risk to allow the actual
true value of the test parameter found outside the
specification when reporting the measurement value exactly
on the specification limit. 15
>50% Risk
above
specification
limit!
Specification-
upper limit
(Maximum)
16. A worked example
• Suppose a test method has a measurement uncertainty (+ 0.35ppm)
of a lead Pb content near its upper feed product specification limit
of 10ppm.
• Upon analysis, a result of 9.65+0.35ppm was found in the feed
sample given with 95% confidence.
• This result indicates that the true Pb content of the sample is
somewhere within the range: 9.30ppm to 10.00ppm with 95%
confidence
• It means that there is 2.5% chance that the true Pb content is
outside both ends of this range. 16
17. The meaning of Measurement Uncertainty
◼ For a measured value of 9.65ppm with an uncertainty of
0.35ppm, we shall write :
◼ 9.65 + 0.35 ppm
◼ The uncertainty range covers :
◼ 9.30 – 10.00 ppm with 95% confidence that the true value will
lie somewhere within:
◼ 9.30 9.65 10.00 ppm
◼ (True Value)
17
=0.025 uncertainty
=0.025 uncertainty
18. A worked example
• Remember that this uncertainty
(U) of +0.35 ppm comes from
• 2 x 0.175 ppm
• where 2 is the coverage factor
for normal distribution at about
95% confidence and,
• 0.175 ppm is the combined
standard uncertainty (u) from
the MU evaluation study.
• To be exact, the coverage factor
should be 1.96 for 95.00%
confidence for two-tailed
distribution situation (+).
By MS Excel functions:
-1.9600 =NORM.INV(0.025,0,1)
1.9600 =NORM.INV(0.975,0,1)
18
19. Use of Excel functions for normal distribution
Excel® entry
-1.960 =NORM.INV(0.025,0,1)
1.960 =NORM.INV(0.975,0,1)
0.975 =NORM.DIST(1.96,0,1,TRUE)
0.950 =NORM.DIST(1.645,0,1,TRUE)
“=NORM.INV(probability, mean, standard_deviation)”
“=NORM.DIST(x, mean, standard_deviation, cumulative)”
Proportion of area under the curve from - to
z- value. When mean = 0 and SD = 1, x = z.
19
z =
z =
20. A worked example
• Hence, if we were to decide on 9.65 ppm as our critical value to
make a “Pass” statement of conformity with confidence, we are
actually taking a 2.5% risk, instead of 5.0%, to be proven wrong if
the same sample were to be tested elsewhere by the same
procedure.
• But we are prepared to take a usual 5% risk in making such
statement.
• What is the critical value if we wish to make a decision rule to
take a 5% risk (or making a 5% error in rejecting the test result)
instead?
20
21. A worked example
• Remember the product
specification’s upper limit is 10.0
ppm Pb
• Our concern therefore is how
much uncertainty of our test
result is in the rejection zone
above the 10 ppm limit
• Hence, the coverage factor should
be chosen at the right-tailed
normal distribution instead of
two-tailed distribution.
• This coverage factor = 1.645,
instead of 1.960 under normal
probability distribution.
By MS Excel function:
-1.6449 =NORM.INV(0.05,0,1)
1.6449 =NORM.INV(0.95,0,1)
21
22. A worked example
• Therefore, the critical value for us to take a 5% risk to reject the result
against the upper specification limit is therefore:
• 10.00 – 1.645 x 0.175 or 10.00 – 0.288 or 9.71 ppm Pb
• That means our decision rule can be stated as below:
• “A pass statement shall be reported when the test result is equal or
smaller than 9.71 ppm Pb content in the feeds with 95% confidence. A
provisional or conditional pass can be reported when the test result is
above 9.71 ppm Pb but below 10.00 ppm Pb. A fail statement is to be
declared for test results above 10.00 ppm Pb. ”
22
23. In summary, graphical effect of the decision rules:
Specification:
10.0 ppm Max
9.65 ppm
9.71 ppm
U = 0.35 ppm U = 0.29 ppm
2.5% Risk 5.0% Risk
Acceptance Zone
Rejection Zone
23
24. Use of Guard Band g with acceptance limits
• ILAC-G8, Eurolab documents suggest the use of Guard
Band g which is the current trend of thinking.
• Guard band (g) is defined as: “interval between a
tolerance limit and a corresponding acceptance limit
where length g = |TL – AL|.”
Upper Tolerance Limit TL
Lower Tolerance Limit TL
Acceptance Limit AL
Acceptance Limit AL
g
g
Nominal
24
25. A graphical example of areas defined for a specification
interval with guard bands
+ U
25
26. Example 1A of areas defined for a specification interval with
guard bands inside the set limits – conservative or stringent
acceptance zone (Type I ) model
26
In this model, the risk taken is 2.5% at = 0.025 only at upper or lower specification limits
27. Example 1B of areas defined for a specification interval with
guard bands inside the set limits – conservative or stringent
acceptance zone (Type I ) model
In this model, the risk taken is 5% at = 0.05 only at upper or lower specification limits
27
28. Example 2 of areas defined for a specification interval with
guard bands outside the set limits – Relax rejection zone
(Type II ) model
28
29. Example 3 of areas created by considering the upper and lower
tolerance limits given by the customer with the laboratory’s
own measurement uncertainty
29
30. Similar approaches for microbiological counts
• Microbiological analysis is based on counting the number of colonies in
terms of CFU (Colony Forming Units) after incubation with appropriate
agar media.
• Discrete counting does not follow normal probability distribution.
• A logarithmic transformation of the counts is to be made in order to
assume a normal distribution for continuous data
• For example, for a count of 100 CFU, Log10(100) = 2.000
• In MU study of microbiological counts, the combined standard
uncertainty is expressed in the form of logarithmic relative standard
deviation, Log10, rel
• Same principles of factor 1.645 are applied like those in chemical analysis.
30
31. The binary
approach
• Applicable for qualitative “Pass/Fail”, “Present/Absent”,
etc. testing
• When you take two samples for testing and if both pass,
you report “Pass”
• If one passes and another fails, then you need to take
two more samples for analysis
• If both pass, it is determined as a pass
• If either or both fail, it fails.
• This approach has implicitly taken uncertainty into
account and is therefore the defined “decision rule”
31
32. In reporting ….
• The laboratory shall :
• apply the decision rule
• Ensure its report to contain either the decision rule used or a reference to a standard
or agreed decision rule
• Listing the uncertainty of measurement result is optional, depending on customer’s
liking
• Making a disclaimer statement, e.g. by ILAC Guide 8.03 “The statement(s) of
compliance with specification (or requirement) is based on a 95% coverage probability
for the expanded uncertainty of the measurement results on which the decision of
compliance is based”.
• There must be enough detail so that if another lab is tasked with taking the same
measurements and using the same decision rule, that they will come to the same
conclusion about a “pass” or “fail”, in order to avoid any legal implication.
32
33. Decision rule documentation, the lab shall ….
33
document the decision rules
employed taking into account the
level of risk associated with the
decision rule employed (i.e. false
accept and false reject) and statistical
assumptions associated with the
decision rule employed;
prepare a list of critical values for the
test parameters with Type I error α =
0.05, i.e. 95% confidence, normally
after some statistical hypothesis
analysis
take time to clearly document what
has been agreed with the clients and
ensure that lab procedures allow for
the correct treatment of the readings
and uncertainties
might need to change test procedures
to cater for the decision rule that will
be applied such as a review of
measurement uncertainty level. It has
to be such that anybody in the lab
will know what is required and what
the limits are.
34. Parting words ….
• Decision rule construction requires four components:
• the measurement result,
• its measurement uncertainty,
• the specification limit or limits, and
• the acceptable level of the probability of making
each type of wrong decision.
• Choice of a decision rule is a business consideration
that takes into account the cost of rejecting an in-
specification product, the cost of accepting an out-of-
specification product, the uncertainty associated with
the measurement, the cost of making the measurement,
and client’s relationship for future business.
34
35. i.e., the lab must agree with the customer on its decision rule and
communicate clearly about the level of risk associated with this
decision rule.
Conclusions
• Issuance of a certificate of conformity or a statement of
compliance allows the laboratory to carry a fair bit of
responsibility.
• The laboratory must be confident on its own measurement
uncertainty estimation.
• If a statement of conformity is required by the clients, a lot
more detail is required to be discussed before carrying out
the laboratory analysis.
• ISO/IEC 17025 requires the Lab to be very clear about the
criteria used and must communicate with and be agreed by
the customer, defined in the test report, certificate of
conformance or statement of compliance to avoid future
disputes.
35
36. Useful References
• ILAC-G8:09/2019 : Guidelines on Decision Rules and Statements
of Conformity
• EuroLab Technical Report No. 01/2017 : Decision rules applied to
conformity assessment
• UKAS LAB 48 Edition 3 (June 2020) : Decision rules and
statements of conformity
• OIML Guide G 19 (2017) : The role of measurement uncertainty in
conformity assessment decisions in legal metrology
• JCGM 106:2012 : Evaluation of measurement data – The role of
measurement uncertainty in conformity assessment
36
