The document discusses lessons learned from building AI models in the financial services industry. It highlights two main challenges - data is often scarce or restricted due to privacy concerns, and data comes from multiple jurisdictions with different regulations. It provides examples of using synthetic data and country-specific iterative modeling to address these challenges. The key takeaways are that rich yet difficult to acquire data is appealing but challenging for AI in financial services, and multi-jurisdiction modeling requires an end-to-end MLOps framework to develop machine learning at scale.

1. Stories from the Financial
Service AI Trenches
Lessons learned from building AI models in EY
18 November 2020
Tim Santos, Assistant Director, Client Technology AI
Mustafa Somalya , Assistant Director , Client Technology AI

2. 18 November 2020Page 2 Stories from the Financial Service AI Trenches: Lessons learned from building AI models in EY
1 AI in Financial Services Overview
2 Use Cases and Learnings
Agenda

3. 18 November 2020Page 3
AI in Financial Services
How does an experiment-driven disruptive technology such as AI look like in a highly-regulated industry?
Sources:
https://www.fca.org.uk/publication/research/research-note-on-machine-learning-in-uk-financial-services.pdf
https://ec.europa.eu/digital-single-market/en/high-level-expert-group-artificial-intelligence
http://rms.koenig-solutions.com/Sync_data/Trainer/QMS/1752-2020328106-AuditingArtificialIntelligencereseng1218(1).pdf
Stories from the Financial Service AI Trenches: Lessons learned from building AI models in EY

4. 18 November 2020Page 4
Review data sourcing, profiling,
processing, as well as data
quality and ethical issues
Assess approach and models are fit for
purpose, explainable, reproducible,
and robust, with supporting evidence
Confirm outcomes achieve desired level
of precision and consistency, and are
aligned with ethical, lawful, and fair
design criteria
Ensure solution is scalable and
deployable with the right tech
infrastructure, and
continuously monitored
Ensure business purpose,
governance and stakeholder
engagement are properly
identified and aligned
Solution
Lifecycle
Modelling
Outcome
Analysis
Deployment
and
Monitoring
Data and
Processing
Business and
Governance
Source: https://www.ukfinance.org.uk/system/files/Trust%2C%20Context%20and%20Regulation%20-%20Achieving%20more%20explainable%20AI%20in%20financial%20services.pdf
Stories from the Financial Service AI Trenches: Lessons learned from building AI models in EY
AI in Financial Services
How does an experiment-driven disruptive technology such as AI look like in a highly-regulated industry?

5. How do you train models for rich
yet highly restricted data that could
be difficult to acquire?
18 November 2020Page 5 Stories from the Financial Service AI Trenches: Lessons learned from building AI models in EY

6. Use Case : Know Your Customer (KYC)
Page 6
KYC
Social Network
Employment
Information
Self-certification
Forms
Biometric Data
Legal Documents
Open
Banking
Proof of Identity
Digital Footprint
KYC requires a lot of time consuming
repetitive manual work that involves
the processing of a variety of data
sources.
Ubiquity, variety of data sources, and
complexity involved in cognitive tasks
make it a very attractive use case for AI.
Stories from the Financial Service AI Trenches: Lessons learned from building AI models in EY18 November 2020

7. Use Case : Know Your Customer (KYC)
Page 7
Form Field Detection
2
Handwritten Text Recognition
3
Data Synthesis
1
Stories from the Financial Service AI Trenches: Lessons learned from building AI models in EY18 November 2020

8. Use Case : Know Your Customer (KYC)
► Data is scarce or highly restricted because of sensitive and personally-identifiable information
► SDLC and DevOps can be inadequate for ML development, consider MLOps
► Treat the scarcity of data as a technological and scientific problem
► When using synthetic or generic datasets, ensure that there’s a feedback mechanism for when live
data becomes available
22 November 2020 Presentation titlePage 8

9. How do you develop models when
data from clients come from
different geographies, have
different legislations and cross-
border restrictions?
18 November 2020Page 9 Stories from the Financial Service AI Trenches: Lessons learned from building AI models in EY

10. Use Case: Multi-Jurisdiction Models
Generic models and pipelines are reused, iterations produce bespoke models by incorporating country-specific data
22 November 2020 Presentation titlePage 10
Reusable Components
Standard ML Pipeline: Base Model
► Common laws and treaties
► Similar industry trends and
treatments
► Transactional trends
► Language models
► Common data model
► Generic dataset
► Regional market
► Cross regional market
► National market
Base Model
Country X Country Y
Model Y v1Model X v2Model X v1
Model X
v3
Model Y v2
retrain
increment
Country Z
retrain
Model Z v1
Model Y v3
MODELS XYZ
ML Pipeline Iteration XYZ:
Bespoke Model
► Hyperparameter Tuning
► Country-specific datasets and
enrichment
► Additional categories and features

11. 18 November 2020Page 11 Stories from the Financial Service AI Trenches: Lessons learned from building AI models in EY
Multi-Jurisdiction Models Example – MLOps and AutoML
Modelling Outcome
Analysis
Deployment and MonitoringData and
Processing
Data Prep
Exploratory
Data Analysis
Feature
Engineering
Feature
Selection
Benchmark
Metrics
Model Serving
(Inference)
Drift
Monitoring
(inference)
Model Build and AutoML Pipeline
Hyperparameter
Tuning
Training
CI/CD
Model Serving
(Train Pipeline)
Retraining/
Rollback /
Increment
Data slicing
Model Serving (Training Pipeline)
Experimentation
Feature
Importance
Drift
Monitoring
(Training)
Model Serving (Inference)
model is stale
make predictions
Human in
the loop
Consume/
Interface
High confidence
Low confidence
Model Drift Monitoring (Data Signature)
model is
good
• Create training (baseline) and inference dataset signatures from features
• Create signatures from predictions, also called theories
• Measure the distance of signatures
• Population Stability Index : 𝑃𝑆𝐼 = ∑!(𝐴! − 𝐵!) ln
"!
#!
{𝐴!, 𝐵! − 𝑓𝑟𝑒𝑞𝑢𝑒𝑛𝑐𝑦 𝑐𝑜𝑢𝑛𝑡 𝑝𝑒𝑟 𝑓𝑒𝑎𝑡𝑢𝑟𝑒 𝑏𝑢𝑐𝑘𝑒𝑡}
• Numerical Parametric (should pass normality, homoskedasticity): T-test
• Numerical Non-parametric: Kruskal-Wallis, Wilcoxon, Kolmogorov, Mann-Whitney-U
• Categorical Features and theory testing: Pearson’s Chi-squared test
• Provide pass/warning/fail logic to trigger retraining, rollback, AutoML, reinforcement learning
Training
Dataset
Inference
Dataset
Model
features
features
predictions
Inference
Signature
Score
(Distance)
Training
Signature
Data
Augmentation

12. Transfer learning and Model Finetuning
18 November 2020Page 12
Use Case: Multi-Jurisdiction Models
Data drift monitoring and MLOps tools
Reproducible end-to-end ML pipelines and AutoML
Leveraging “human in the loop” with MLOps framework
and online learning
Enabling components for Multi-Jurisdiction and ML at scale
Time from Technical and Business
SMEs are valuable, a complementing
operating model and tooling would
be necessary to maximise value
Building and deploying bespoke
models for each jurisdiction is difficult
to scale without an end-to-end
MLOps platform
Stories from the Financial Service AI Trenches: Lessons learned from building AI models in EY

13. 18 November 2020Page 13
The appeal of using AI in FS lies in very rich data,
the same reason that makes data very challenging
to acquire.
AI in FS usually involve clients in multiple
jurisdictions, it is imperative to have MLOps
framework and platform to develop ML at scale.
Stories from the Financial Service AI Trenches: Lessons learned from building AI models in EY
Key Takeaways

14. Q&A
18 November 2020Page 14
Tim Santos
Assistant Director | Global IT
► Global Client Technology AI
► MLOps Lead
► Timothy.Santos@uk.ey.com
Mustafa Somalya
Assistant Director | Global IT
► Global Client Technology AI
► ML Experimentation Lead
► Mustafa.M.Somalya@uk.ey.com
Stories from the Financial Service AI Trenches: Lessons learned from building AI models in EY

15. EY | Assurance | Tax | Strategy and Transactions | Consulting
About EY
EY is a global leader in assurance, tax, strategy, transaction and consulting
services. The insights and quality services we deliver help build trust and
confidence in the capital markets and in economies the world over. We develop
outstanding leaders who team to deliver on our promises to all of our
stakeholders. In so doing, we play a critical role in building a better working world
for our people, for our clients and for our communities.
EY refers to the global organization, and may refer to one or more, of the member
firms of Ernst & Young Global Limited, each of which is a separate legal entity.
Ernst & Young Global Limited, a UK company limited by guarantee, does not
provide services to clients. Information about how EY collects and uses personal
data and a description of the rights individuals have under data protection
legislation are available via ey.com/privacy. For more information about our
organization, please visit ey.com.
This news release has been issued by EYGM Limited, a member of the global EY
organization that also does not provide any services to clients.
© 2020 EYGM Limited.
All Rights Reserved.
EYG no.
ED MMYY
This material has been prepared for general informational purposes only and is
not intended to be relied upon as accounting, tax or other professional advice.
Please refer to your advisors for specific advice.