Model Risk guidance was provided by the OCC and FED in 2011. We comment on their proposal, review other relevant literature, and conclude with an user friendly checklist.

1. White Paper:
Model Risk
Version: Final
September 22, 2011

2. 2 Introduction
Table of Contents
New Supervisory Guidance on Model Risk Management ........................................................... 2
Introduction ............................................................................................................................ 3
Definitions .............................................................................................................................. 3
Model Usage .......................................................................................................................... 4
Better Practices Guidance ...................................................................................................... 5
Model development, implementation, and use ........................................................................ 5
Development ....................................................................................................................... 5
Implementation ................................................................................................................... 6
Use ..................................................................................................................................... 6
Model verification and validation ............................................................................................. 6
Model Risk Governance ......................................................................................................... 7
A Note on Third-Party Models ................................................................................................ 7
Self-Assessment of Your Model Risk Management Practices................................................. 8
Conclusion ............................................................................................................................. 9
Authors ................................................................................................................................... 9

3. 3
White Paper: Model Risk
Introduction
In early April 2011, the Office of the Comptroller of the Currency (OCC) and the Board of
Governors of the Federal Reserve System (FED) released Supervisory Guidance on Model Risk
Management (the Guidance or OCC 2011-12). From our perspective, there was little new in the
guidance. However, OCC 2011-12 codifies better practices that were previously familiar only to
sector specialists, including the Capital Markets teams at the OCC and FED. So the news is not
that there is anything new, per se, rather the news is that better practices have been clearly
communicated to OCC- and FED-supervised banks and thrifts via OCC 2011-12 i.
The Guidance reinforces prior regulatory recommendations. From a regulatory perspective, this
guidance was issued to “provide comprehensive guidance for banks on effective model risk
management” ii. An unwritten cause of this issuance was that bank supervisors have observed
substandard model risk practices across the industry at institutions both large and small.
The Guidance notes that the use of models, especially complex models, varies across the
industry, so model risk management and mitigation should be commensurate with risk and
model complexity. As is typical of regulatory issuances, it notes that internal controls, policies,
and documented procedures are appropriate.
The FDIC and NCUA were not included in this issuance, as they were with the Interagency
Advisory On Interest Rate Risk Management (OCC 2010-1a) and the Interagency Policy
Statement on Funding and Liquidity Risk Management (OCC 2010-27a). As with OCC 2000-16
(Model Validation) and SR 09-1 (Application of the Market Risk Rule in Bank Holding
Companies and State Member Banks), it is likely that FDIC and NCUA supervisors will apply
OCC 2011-12 to supervised institutions.
Definitions
A model is defined as “a quantitative method, system, or approach that applies statistical,
economic, financial, or mathematical theories, techniques, and assumptions to process input
data into quantitative estimates”. The definition of model is expanded to include “quantitative
approaches whose inputs are partially or wholly quantitative or based on expert judgment,
provided that the output is quantitative in nature”.
The Guidance, congruent with OCC 2000-16, delineates three components of a model:
1. An information input component, which delivers data and assumptions to the model
2. A processing component, which transforms input into estimates
3. A reporting component, which transforms estimates into useful business information.
The Guidance suggests there are two main sources of model risk:

4. 4 Model Usage
1. Use of a model with errors, including:
a. an incorrect theoretical foundation
b. improper implementation or maintenance
c. inaccuracies in any of its three components (as above).
2. Incorrect or inappropriate use of a model
a. Use a model in inappropriate circumstances
b. Results that are not decision makers
c. Lack of explanation or understanding of limitations
The Guidance defines model risk as “the potential for adverse consequences from decisions
based on incorrect or misused model outputs and reports”. This is a good summary definition as
it also includes the potential for poor decisions based on the outputs of technically correct, fully
validated models that were not totally understood by decision makers (e.g. Value at Risk
models, 2. above).
Eugene Derman’s 1996 article iii on Model Risk provides more information on the types of model
risk:
• Wrong model
o Inapplicability of model
o Incorrect model specification
• Model implementation
o Programming errors
o Technical errors
o Use of numerical approximations
• Model usage
o Implementation Risk
o Data issues
o Calibration errors
Two of the primary tools for effectively managing model risk are model verification and model
validation iv. Model verification refers to a process aimed at answering the question “did we
build the model right”. Model validation refers to a process aimed at answering the question
the question “are the results credible?
Model Usage
Industry observers and the Guidance recognize that the use of models in banks is pervasive in
pricing, risk management, and reporting:
• evaluate and underwrite credits
• price loans and deposits
• create and analyze forecasts and strategies

5. 5
White Paper: Model Risk
• measure and monitor credit, liquidity, market, and operational risks
• value exposures and financial instruments
• report and disclose results
The Guidance states, “The use of models invariably presents model risk”. Derman noted, “This
reliance on models to handle risk carries its own risks”. The authoritative Verification, Validation,
and Accreditation of Army Models and Simulations v simply say, “There is no such thing as an
absolutely valid model”. Accordingly, we would suggest that best practices model risk
management practices recognize that model risk is unavoidable, but that effective model risk
management mitigates model risk within a cost/benefit context.
Better Practices Guidance
The Guidance highlights three areas for effective, best practices risk model management:
• Model development, implementation, and use
• Model validation
• Governance, policies, and control
While the above framework is not new to bankers, the Guidance notes “A guiding principle for
managing model risk is "effective challenge" of models, that is, critical analysis by objective,
informed parties who can identify model limitations and assumptions and produce appropriate
changes.”
The notion of “effective challenge” is perhaps new to all but the largest of banks. In very large
banks, a specialized Model Risk or Model Validation unit reports to the Chief Risk Officer. In
most banks, Internal Audit staff or the Chief Financial Officer’s team is tasked with model
review. Usually, these units lack the expertise necessary to review the theoretical underpinnings
and math of complex models. As a result, the review is ineffective, and frequently relies solely
on checklists emphasizing balancing to financials and policy compliance, which while necessary
are not sufficient.
Model development, implementation, and use
This topic is broadly consistent with the definition of model verification, as defined above.
Examples from the ALM model sector may illuminate the notion of “effective challenge” in this
area.
Development
As regards vendor model development and documentation, we have found that some models,
even those with hundreds of client banks and that have been “certified” by a third party, have

6. 6 Model verification and validation
fundamental valuation flaws. The flaws have been noted in both the valuation model math and
in the documentation of how to implement the valuation tools. Perhaps this is due to the greater
subject expertise of the model verification and validation team than the model development or
certification teams. In any event, it is unlikely that community bank Internal Audit teams have
sufficient expertise in valuation models to “effectively challenge” the use of a vendor valuation
model or module, so the use of outside experts is important.
Implementation
We have also observed inadequate implementations of otherwise sound risk models. Recently,
at a $1+ billion National Bank, we asked the ALM staff about the model implementation and
subsequent model calibration. The staff readily noted that they had not updated its pricing,
valuation, and prepayment configuration, including assumptions, over the past ten years since
initial vendor implementation “because the model did it automatically”. Unfortunately, or
fortunately, depending on your perspective, for the bank and readers of this publication, the
state of the art over the past decade is such that no ALM model automatically updates every
assumption appropriately and automatically.
Use
One of the key assumptions in the use of an ALM model is Core Deposit rate sensitivities. In
many banks, unsubstantiated and undocumented rate sensitivity estimates are used. This has
the impact of suggesting that their bank is either “asset and/or liability sensitive”. The use of
alternative assumptions, via sensitivity analysis, market or institution statistical studies, or other
quantitative approaches may affect the magnitude and/or directionality of bank risk exposures.
Again, the typical Internal Auditor is unfamiliar with the risk reporting consequences of key
assumptions, let alone the impact such changes may have on product pricing, hedging
strategies, and balance sheet allocation decisions.
Model verification and validation
The Guidance outlines three core elements of an effective validation:
1. Evaluation of conceptual soundness and assessment of the quality of the model design
and construction
2. Ongoing monitoring, including process verification and benchmarking
3. Outcomes analysis, including back-testing

7. 7
White Paper: Model Risk
The topic of Model Validation has been covered in previous BALM articles, including ours, and
will likely be the topic of futures articles, including ours. However, we would note that models
are used in other parts of the world in addition to banking and present an example vi of
verification and validation activities. Accordingly, we suggest that “model validation” as used in
Banking typically refers to “model verification and validation” in other sectors.
Model Risk Governance
To control model risk, banks need a strong, thorough model governance process. This includes
outlining the important roles for the Board, senior management, the model operators/owners
and the model validators. The key is defining a workable structure that ensures the integration
of high quality model forecasts with management decision making and regulatory requirements.
The challenge is to do this in a way that does not create a system crippled by
micromanagement processes.
The following summarizes key requirements for effective governance:
• Board of Directors and Senior Management - establish a strong model risk management
framework that fits into the broader risk management of the organization
• Policies and Procedures – require model risk management activities be formalized
through policies and documented procedures to implement them
• Roles and Responsibilities – address ownership, controls and compliance
• Internal Audit - assess the overall effectiveness of the model risk management
framework
• External resources – specify in writing third-party resources engaged to assist with
model validation and review, compliance functions, or other activities in support of
internal audit
• Model inventory – require the maintenance of a comprehensive set of information for
models implemented for use, under development for implementation, or recently retired
where applicable
• Documentation – maintain detailed documentation of model development and validation
so that parties unfamiliar with a model can understand how the model operates, its
limitations, and its key assumptions.
A Note on Third-Party Models
Some banks rely on vendors to run financial models and prepare risk reports for them. The
guidance addresses the validation of vendor and third-party products by stressing that although
the process may be somewhat modified, the validation principles should be the same as applied
to in-house models. This helps bank management better understand the vendor product and its
capabilities, applicability, and limitations.

8. 8 Self-Assessment of Your Model Risk Management Practices
The Board and bank management should ensure that the model works effectively if they use
reports to make decisions. Management should be diligent in tracking changes the vendor
makes to the model including for each reporting period a list of any model changes and an
explanation of the expected effects on the resulting reports.
Self-Assessment of Your Model Risk Management Practices
Developing, implementing, and maintaining an effective model requires ongoing diligence in
oversight, validation and controls management. Ask yourself, or have your Internal Audit ask
these questions for each model your bank uses:
1. Are the model results significant to managing your bank’s financial position and/or
defining future strategy?
2. Does the model have a sound theoretical foundation?
3. Are the assumptions reasonable and supported?
4. Do senior management and the Board understand the model’s primary purpose,
methods, and calculations?
5. Are model limitation regularly communicated and understood?
6. Are the data inputs accurate?
7. Are model results translated into understandable analytical reports that support decision
making?
8. Has your model been validated within the last twelve month period?
9. If your model has been validated, have significant findings and recommendations been
addressed?
10. If the model is a third-party model, then answer these additional questions:
a. Do you have an appropriate process in place for selecting vendor models?
b. Do you require that the vendor provide developmental evidence explaining the
product components, design, and intended use, to determine whether the model
is appropriate for your bank’s products, exposures, and risks?
c. Does your vendor regularly provide appropriate testing results, showing their
product works as expected?
d. Is your bank clearly aware of the model’s limitations, detailed assumptions and
where the model’s use may be problematic with respect to your particular
products?
e. Does your vendor regularly report on and conduct ongoing performance
monitoring and outcomes analysis, and make appropriate modifications and
updates over time?
f. Are your bank’s model customization choices documented, justified and
understood?
g. If your model vendor provides input data or assumptions, are you provided with
details and is the relevance to your bank’s situation understood and assessed?
h. Are you conducting ongoing monitoring and outcomes analysis of vendor model
performance using the bank’s own outcomes?

9. 9
White Paper: Model Risk
i. Do you have an in-house model expert who understands the systematic
modeling procedures that can help the bank understand the vendor product and
its capabilities, applicability, and limitations?
Conclusion
The recent Guidance alerts banks to the emphasis that the OCC and FED are placing on
effective model risk management. Over the past two years, we have seen an increase in formal
and informal regulatory orders related to effective model risk management for market, credit,
and liquidity risk models. These orders have also noted the effectiveness and frequency of
Board, ALCO, and Credit Committee reporting. We suggest that you use the “Self-Assessment”
above and review the Guidance as time allows.
Authors
Fred Poorman Jr., CFA, Managing Principal, Bank Risk Advisors
Howard Stern, PhD, Senior Consultant, Bank Risk Advisors
Terry Treadwell, CPA, Senior Consultant, Bank Risk Advisors
i
Existing regulatory guidance includes– the FDIC’s Supervisory Insights (Winter 2005) article on Model
Governance; OCC Bulletin 2000-16, the current guiding directive on ALM model verification for national
banks, as well as mandates outlined in various other regulatory directives -- the most recent IRR advisory,
2010-1A, and discussion in FDIC's Supervisory Insights (Winter 2009).
ii
Unless otherwise noted quotations are from the Guidance
iii
http://www.ederman.com/new/docs/gs-model risk.pdf
iv
http://ppc.documents/model-verif-validation-full.pdf
v
http://www.apd.army.mil/pdffiles/p5 11.pdf
vi
http://ppc.documents/model-verif-validation-full.pdf