Applied AI lecture for NTU MBA class. Discussion of better ways to understand learning technologies (AI) and discussions around Enterprise considerations for Learning Algorithms including Fairness Ethics Accountability Transparency (Explainability).

1. INTRO TO AI
APPLYING
LEARNING
TECHNOLOGIES TO
THE ENTERPRISE
LUIS F GONZALEZ
GUEST LECTURER
MD ELEMENTAI ASIA PACIFIC

2. SLIDO –
Polling the class:
1. In which area AI going to impact your life the
most in the next 20 years?
2.What topic of AI are you most interested to
learn?
3. What worries you the most about AI in society?

3. Agenda
Intro to Learning Algorithms (AI)
The Learning Enterprise
XAI – Human Experience and FEAT
Group Discussions

4. What are Learning Technologies?
The theory and development of
computer systems able to perform
tasks that traditionally required human
intelligence like learning. Artificial
Intelligence is somewhat inaccurate as
systems are not intelligent alas, they
learn.
Video must see to understand deep learning
https://youtu.be/5tvmMX8r_OM

5. Let’s simplify the progression
Input
Input
Input
Hand-designed
rules
Output
Learned simple
features
Learned complex
features
Mapping from
features
Output
Hand-designed
features
Mapping from
features
Output
Rules-based systems
1960’s
Classic machine learning
2000’s
Deep learning
2020’s
Input
Random
Probability
Distribution
Output
Stochastic systems
1980’s

6. Types of Learning Possible

7. Hierarchical Abstractions

8. What are Learning Technologies Components?
Sensing the world
Perception
Learning from every
interaction
Communication
Optimizing to specific
outcomes
Decision making
Understanding
concepts & relations
Reasoning
Taking actions in the
world to achieve goals
Interaction
Computer Vision
Natural Language
Understanding & Generation
Forecasting and Operations
Research
Knowledge Graphs
and Representations
Reinforcement Learning
Answer questions about a scene
Determine if a growth is cancerous or not
Infer what happened to
characters in a story
Drive on city streets and highways
Identify objects in a scene

9. Perception: Vision and its applications

10. Reasoning: Knowledge
Graphs &
representations

11. Communication: Natural Language

12. Applications today For NLP/G/U
Abstract Summarization helps us understand your Context:

13. Decision: Time
Series and its
applications

14. Interaction: Reinforcement Learning
Reward Functions

15. Agenda
Intro to Learning Algorithms (AI)
The Learning Enterprise
XAI – Human Experience and FEAT
Group Discussions

16. Normal Components of AI in Enterprises
5
Object Detection
& Monitoring
Robust detection, counting and tracking of
objects and people in a wide variety of
environments, enabling valuable
workflows in many real-world situations.
Image Classification
Video Alteration
Object Tracking and Counting
6 Optimization
AI-powered optimization boosts
the efficiency of business processes and
tasks, maximizing lift and ROI compared to
traditional optimization methods.
Routing
Disruption management
Re-optimization
7 Explainability
Making your AI explainable to users helps
drive adoption and lowers the barrier of
entry for users.
Technical Explanations
Bias Evaluation and Tracking
Sample-based Explanations
And more including...
● Recommender systems
● Assignment with constraints
● Association rule learning
● Human-AI interaction
● Image segmentation
● Image clustering
● Routing with constraints
● etc..
4 Time-Series Forecasting
Best-in-class AI-powered forecasting
provides better accuracy and can deliver
lift for a wide range of forecasting
scenarios.
Hybrid Forecasting Models
Statistical Forecasting
Deep-learning Forecasting Models
3 Anomaly Detection
Detect anomalies on objects in
natural environments and in various types
of data, allowing for near real-time
reaction.
Visual Anomaly Detection
Event-based Anomaly Detection
Anomalies in Forecasting Data
1 Text Extraction & Analysis
Accelerate the extraction of insights from
multiple forms of text, catching signals
that are easily overlooked by humans.
Text Summarization
Sentiment Analysis
Text Classification
2
Optical Character Recognition
Instantly transcribe text from natural
environments or digital documents,
reducing manual work and enabling
automation.
Documents
Handwritten Notes
Live Scenes and Video

18. Learn user’s behaviour based on voice commands
and can adjust settings automatically in
subsequent interactions
Target user with personalized products and
services ads based on their demographic profile,
search history, visited sites, liked social media
posts, etc.
Improve efficiency/quality in servicing customers by
using AI assistants (e.g., chatbots, robo-greeters in
bank branches and cardless ATM machines via facial
recognition)
Detect suspicious/fraudulent activities in network
and/or transactions using predictive analytics
Recommend music or videos based on user’s
historical consumption and preferences
Provide best driving routes, ETA, and/or match
drivers with riders based on historical and real-
time data
Smart Home Devices Media & Entertainment Navigation and Transportation
E-commerce & Targeted Ads Customer Service Assistants Security and Fraud Detection
Examples - Applications by Industry

19. Element AI | Copyright © 2019 | Strictly Confidential
Typical patterns of AI adoption in enterprise & associated challenges
Major challenges in enterprise AI adoption stem from lack of well-designed AI strategy and enablement support
REPRESENTATIVE
Strategy & Enablement Experimentation Production deployment
No effort in AI strategy/roadmap Algorithm-focused tests in silo
Lack of basic capabilities to run
enterprise AI solutions
Bottom-up use case collection &
arbitrary prioritization
AI roadmap with traditional
methods, part of digital strategy
Various experimentation themes
(technical + non-technical)
Lack of investment in integrated AI
architecture for scale
Lack of investment in enterprise
deployment infrastructure
Algorithm-focused tests in silo
• No buy-in from leadership
• No strategy & capabilities definition
• Limited use cases
• No end-to-end consideration for real application
environment
• Prone to resource overuse w/o result
• Failure in deployment of tested models (technical
& business reasons)
• Short-sighted, low hanging fruit only
- difficulties in proving value of AI
• Misinformed decision on AI strategy &
capabilities acquisition
• Lack of architectural consideration
• Misled investment priorities in experimentation • Difficulty in proving ROI of AI solution
• Lack of productivity in architecture
• High exposure in risk, bias
• Difficulty in proving ROI of AI solution
• No enterprise-grade infrastructure
• High exposure in risk, bias
• No experimentation strategy
• Missing experimentation other than algorithm
performance
Pattern 1.
Experiments
w/o
plan
Pattern 2.
Bottom-up
planning
Pattern 3.
Incapable of
‘scalable AI
adoption’

20. Personas in the Learning Organisation
AI Strategy
Strategy Lead
Learning Processes Owner
Data & Learning Strategist
Intelligent Business Strategy AI Build &Deploy
Data & Learnings Engineer
Applied Research Engineer
AI Developer
AI Discovery
Business Analyst
Data & learnings Analyst
Applied Research Science
HCI Designer
Data Stewart
AI Integration
Learning Algorithms Tester
ML/DL Trainer
AI Operations
Learning Algorithms
Application Support
Performance Monitor
Learnings Stewart
AI Management
Procurement Lead
Learning Tech Governance Lead
AI Legal Specialist
Ecosystem Relations
AI Infrastructure.
Platform Lead
Microservices & Learnings Architect

21. What is the most important skillsets in an
Learning (AI First)Enterprise?

22. Learning Technologies Process Framework
AI Strategy
AI StrategyFormulation
AI Governance
AI Research & InnovationAI
Architecture & Design
Business Demand &Impact
Mgmt
AI Use-case portfolio
planning andscheduling
AI Build &Deploy
SolutionPlanning
Solution Build
Solution Test
AI Discovery
AI ProblemDefinition
Data Analysis
Create
Test
Learn
AI Integration
AI ChangeIntroduction
AI ChangeAdoption
AI ChangeManagement
AI ChangeExecution
AI Operations
AI Service Operations
AI Continuous Learning
AI Upgrades
AI Budget
AI ContractsManagement
AI Management
Procurement
AI Governance and RiskManagement
Model Performance Management
AI TalentManagement
AI Infrastructure

23. Element AI | Copyright © 2020 | Strictly Confidential

24. Learning Organization in a page
RaaS
Ready made products Bespoke solutions
LEARNING TECHNOLOGIES ENGINEERING & INTEGRATION
LT MODEL TRAINING AND
DEVELOPMENT
Build Deploy Run Improve
ECOSYSTEM & PARTNERSHIPS
ACADEMIC RELATIONS
Research Led Innovation &
Though Leadership
AI PLATFORMS
LEARNING TECHNOLOGIES RESEARCH INNOVATION
FUNDAMENTAL RESEARCH
Publications
Innovation Fundamentals
APPLIED RESEARCH SCIENCE
Model Applications
Industry Innovations
Startup
Incubation &
M&A
LT VENTURES
Consultation and
Advisement
REGULATION
LEARNING TECHNOLOGIES GOVERNANCE FRAMEWORK
LEARNING
TECHNOLOGY SUPPORT
LT Integration and
Geographical support
LT Sustainment &
Continuous
Improvement
LT SOLUTIONS ENABLEMENT
Use Case on New Revenue
Stream Gen
Use Case on Cost Reduction
LEARNING TECHNOLOGIES OPERATIONS & CONTINUOUS
IMPROVEMENT
Learning
Technologies
Outsourcing
Solutions
Agile, Rapid prototyping and
business model adaptation
LT Ideation and Design
thinking
LT METHODOLOGY
OUTSORCING
OPERATIONS
OS
Hypothesize Design Collect Assess
Data
Model Prototype Deploy Operate Maintain Improve
DATA SCIENCE & EGINEERING DATA MODELLING & MANAGEMENT
CO-Creation and Solution
Synergies
CO-DEVELOPMENT
Talent Pipeline &
Education
RESOURCES
Hypothesize
HUMAN RESEARCH
COGNITIVE, BEHAVIOURAL AND
CULTURE
Use Case on Improvement
ML DEV,
MULTICLOUD, EDGE &
DATA
PLATFORMS
Research
Engineering
Advisory/Enablement

25. Structured Data Unstructured Data Semi-Structured Data
Description
• Formatted data set with
defined fields such as
(numeric, currency,
alphabetic, name, date,
address, etc.,)
• Generally, stored in
relational database
format, tables, excel,
CSV, etc.,
• Often managed using
Structured Query
Language (SQL)
• Sources of information
that cannot easily be
classified into relevant
data fields
• Generally, sources of
information are
classified between
Textual vs. Non-Textual
data
• Textual: PDFs, Books,
etc.,
• Non-Textual: Images,
Audio, Video
• A cross between
Structured and
Unstructured Data
sources
• Generally, tags or other
types of markers are
used to identify
specified elements
within the information
sources, but lacks a rigid
structure
Examples
• CRM purchase history
• Historical stock price
data
• Cellphone Metadata
• Financial Reports
• Text from a PDF
• Satellite Images
• Music audio file
• Security camera video
file
• Database of images
with tagged data fields
such as data, creator,
location, keywords
describing image
Data Realities

26. 23
AI Infrastructure – Flexibility introduced to Enterprise Architecture
High level considerations
ILLUSTRATIVE

27. Element AI | Copyright © 2019 | Strictly Confidential
Plan for enterprise-grade AI architecture & processes
AI adoption at the core of enterprises requires ‘industrialized’ architecture/processes for productivity and sustainability
Use case design
Data collection &
preprocessing
Model building
/selection
Testing &
deployment
Finding insights &
communications
45~60
%
20~25
%
10%
3%
Integrated process
management
10%
Effort Distribution in Typical Data Science Projects (% of time)
• Manual data handling
• Lack of active-learning in
data handling tools
• Lack of toolchain integration to automate data-model-deployment procedures, hindering
productive ML application development
• Mere collection of open-source frameworks without designing reusable, integrated, and
scalable ML application development architecture
• No use case repository
• Siloed use case
management
REPRESENTATIVE
• Manual, ad-hoc collection/sharing of learnings
• Lack of architectural, process embedding for AI risk &
explainability
• No application
framework for model
reuse, cross-learning
Hypothesize Design
Collect Assess
Data
Model Prototype Deploy Operate Maintain Improve Hypothesize
Research
Engineering
Advisory/Enablement

28. The Learning Organisation Kickstart

29. Build a 3 minute pitch on how you propose the
company you work in build a Learning
Technologies capability (COE, SideCar, M&A, ETC.)

30. Agenda
Intro to Learning Algorithms (AI)
The Learning Enterprise
XAI – Human Experience and FEAT
Group Discussions

31. Agenda
Intro to Learning Algorithms (AI)
The Learning Enterprise
XAI – Human Experience / Fairness and Transparency
Group Discussions

32. Fairness Ethics Accountability Transparency
FEAT principles were created
to guide better AI development
Justifiability
Accuracy & Bias
Internal &
External
Explainability
Interpretability

34. Fairness Ethics Accountability Transparency
Sources: Bill Rankin, “Radical Cartography”
http://www.radicalcartography.net/index.html?chicagodots; David Ingold and Spencer,
“Amazon Doesn’t Consider the Race of Its Customers. Should It?”; Calvano, E, G
Calzolari, V Denicolò and S Pastorello (2018a), “Artificial intelligence, algorithmic pricing
and collusion,” CEPR Discussion Paper 13405;
Bias Unintended
Consequences
Systemic
Issues
1 2 3

35. XAI for Trust
● How do I know the model is doing something
reasonable?
● High-stakes decisions
● Counter-intuitive outputs
End users who are responsible for high-stakes
decisions will not blindly trust an algorithm without
what they believe to be strong evidence that its
recommendations or outputs are accurate. One
powerful way to build this trust is if the algorithm
provides an explanation along with its output.
Strong evidence on the
accuracy of the output for high-
stake decisions.
1. Need to know why model is
recommending this
potentially risky/costly
decision?
1. Why model output is
counter-intuitive?

36. XAI for Compliance
Right to an explanation if
receiving an adverse decision on
a loan or insurance policy

37. XAI for Bias
Ensure that people are not being unfairly excluded
Ensure you are not missing a key part of the market when
identifying potential customers

38. XAI for Generalization
Validate the reasoning continuously matches
the business objectives, prevent concept
drift (objectives, data, how close you are test
time from your training distribution)

39. Future of the Learning Organizations
39
TIME
VALUE
Current Tech Org
Future Learning Organizations
New business model
Incumbent’s business
model
Focus on Adoption (advertisement)
Exploitation of Data
Humans as Research Subjects
Harvesting of existing Data
"Context Collapse"
Results > Matrix
Focus on Value Co-creation
(Economic Communities)
Empowerment of Humans
Predictions with you and
Transfer/Active Learning
F.E.A.T. = Trust
Results > Star Trek
Learning Tech
Enablement
Prediction > Personalisation > Humanisation

40. Examples of Fairness
High-
frequency
trading
Accuracy needed
Trust needed
Auditing of
financials and
controls
Diagnostic
medicine
Marketing
spend
optimization
Document
search for
litigation
Maintenance
recommendati
on
Examples of Explainability
Explainability is about communication with a
human and so has much in common with design
and education.
In particular there is no established metric to
evaluate the quality of an explanation, so tracking
progress and comparing methods is hard. In some
cases there may not be a metric.
The field is very immature. There are not yet
standard approaches, benchmarks, or indeed
definitions of the field.
Interpretability
#1 This is why I refused the application
“I refused the application because variable X_2 was
too low and this explained 60% of the variance
considering that X_4 was high and X_7 account for
more than 15% of the decision”
#2 This is why I believe the image is a truck
Many published methods have not actually been tested
with human subjects to validate how math functions
translate to user outcomes
Source: https://arxiv.org/abs/1907.07374

41. User roles spectrum
● End-users: input-output
relationship
● Developers: inner working
of the model
● In practice, the desired
explanations are a
combination of these two
extremes
Application domain
● Some industries might
already have specific
requirements, e.g. banking
and insurance
Explainable AI needs to be designed with
the user role and the application domain in mind

42. Explainability is about communication
with a human and so has much in
common with design and education.
In particular there is no established
metric to evaluate the quality of an
explanation, so tracking progress and
comparing methods is hard. In some
cases there may not be a metric.
The field is very immature. There are
not yet standard approaches,
benchmarks, or indeed definitions of
the field.
An example metric
How much does the explanation improve a user’s ability to predict
future model outputs?
Why is explainability hard?

43. Going beyond model debugging
What kind of action can we drive?
Actionable XAI
Explanations that drive actions
“I need to understand why a model made
a decision so I can complete a regulatory
audit report”
Human-AI Interaction for XAI
AI can interact and also learn from human
insight
“I want to easily see the reasoning so I can
correct errors and give better feedback on
what I want the output to be”
End to End XAI
Moving away explaining a model to
explaining a business process.
“If I am assembling a car, understanding
every piece separately is not sufficient. I
must be able to audit the assembly
process.”
1 2 3
44

44. Case Study at a Aisin Seiki by explaining model errors
CASE STUDY
The manufacturer needed to explain accuracy, confidence in
predictions, and in- and out-of distribution operating range for
plant operators and customers to trust the results.
We implemented explainability components on top of the
client’s existing model to identify negative (erroneous)
influences in the training data and re-calibrate its confidence
estimate to minimize misclassifications.
Early results showed that false positives could be reduced by a
factor of 10X while maintaining nearly 0 false negatives with the
more explainable workflow.
A leading auto parts manufacturer wanted to
automate quality inspection with machine
vision. A multi-XAI component workflow helped
identify and reduce errors.
45
Decision uncertainty
Feature attribution
Sample-based
explanation

45. We believe the way forward for explainability is
an end-to-end approach that requires combining many techniques
ELEMENT AI PERSPECTIVE
Data collection
What data points matter
most? Where might there
be gaps or issues in
representativeness?
Model selection
Can complex models be
constrained or simplified
to be more interpretable?
EXPLAINABILITY IN THE AI APPLICATION
DEVELOPMENT LIFECYCLE
Training & evaluation
What is the model actually
learning? How well will it
perform for different groups
and scenarios?
Interface design
What do different user
personas understand?
What information do they
need to trust it?
Most open-source tools and
platforms focus narrowly on
explaining models (such as
which features are important)
Performance and trust are affected by the whole application development lifecycle. Like supply chain quality
assurance, AI applications need to trace the impact of errors forward to affected clients and backwards to the
source.

46. Learning Technologies Future is XAI enabled
Level 1 Efficiency Level 2 Personalization Level 3. Reasoning Level 4 Exploration
Definition Learning Technologies facilitates
specific functions with systems
and devices, making user
interactions
more efficient and effective
AI uses pattern learning
to recognize, optimize and personalize
functions in order to improve and simplify
interactions for users
AI uses causality learning
to understand the cause
of certain patterns and behaviors, this
information is used to predict and
promote positive outcomes for users
AI uses experimental learning to
continuously improve, by forming
and testing hypotheses it uncovers
new inferences, seamlessly adding
value to users’ lives and enabling a
deeper affinity
Presence
in our lives
Familiar: Systems and devices that
utilize AI are appearing in user’s
everyday lives
Common: AI is optimizing most devices at
the edge and most systems through the
cloud
Universal: AI is everywhere and
interconnected for the benefit of all
devices and systems
Foundational AI forms a core
component of the infrastructure for
all devices and systems in society
which share and learn collectively
Human
Experience
Task Centric: Can execute specific
commands within specific
parameters
to achieve a specific task.
Humans need an understanding of
how models operate in relation to
the task to achieve proficiency in
their use and manage their
limitations.
Goal-oriented: Multiple actions
Works out various options for achieving a
given goal and presents them to the user
for selection or is pre- programmed to
efficiently meet the desired goal. User
needs an understanding of the tactical
outcomes to better perform and trust the
models interacting.
Mission-focused
Long-term actions
Understands users and its
environment in order to predict,
recommend and execute solutions to
assigned missions. User needs
expertise in the strategic direction and
involves in the coordination of efforts
to best use these models.
Purpose-driven
Exploratory actions
Using local context and external
sources of knowledge, it balances
users’ competing needs and
interests and is able to take
creative approaches to influence
user behaviors, whilst in service of
the user’s higher purpose. User
helps orchestrate, prioritize and
participate at a peer to peer level in
a huma and machine teams.

47. Appendix:
Follow UpMaterial
1.Data Trusts New tool ForGovernance. https://hello.elementai.com/rs/024-OAQ-547/images/Data_Trusts_EN_201914.pdf
2. Applying AI Building and Organization for scalingAI: https://techwireasia.com/2021/02/why-scaling-ai-is-an-enterprise-grade-challenge/
4. Building the AI-Powered Organization Harvard Business Review https://hbr.org/2019/07/building-the-ai-powered-organization
3.Enterprise Software The case for Services in Enterprise Software Startups. https://a16z.com/2018/03/12/services-enterprise-software-
products-startups/
5. Michael I Jordan Lex Fridman Podcast. https://youtu.be/EYIKy_FM9x0