I will first survey how deep learning has disrupted speech and language processing industries since 2009. Then I will draw connections between the techniques for modeling speech and language and those for financial markets. Finally, I will address three unique technical challenges to financial investment.
SQL Database Design For Developers at php[tek] 2024
Li Deng at AI Frontiers : From Modeling Speech/Language to Modeling Financial Markets
1. AI Frontiers Conference, San Jose Convention Center, Nov. 9-11, 2018
From Modeling Speech/Language to Modeling Financial
Markets
Li Deng, Chief AI Officer
November 9, 2018
2. Outline Of The Main Topics
1Will AI transform the financial
markets?
Speech
Computer vision
NLP
Robotics
…
…
Finance
2Three technical challenges
unique to financial investment
industry
3Other constraints in applying AI
to financial investment
management
3. Will AI Transform The Financial Markets?
What can we learn from successful AI applications in other industries:
AI disrupting speech industry (2009-present)
– (Small) similarities to finance industry
– (Large) differences from finance industry
AI disrupting computer vision industry (2012-present)
AI disrupting NLP (2014-present)
Learning From Other Industries
4. Launch of Deep Learning
in Speech at NIPS in 2009
Disrupting The
Speech Industry
5. Disrupting The
Speech Industry
Deep Learning practically
solved the speech recognition
problem by 2012 By John Markoff
Tianjin, China, October 25, 2012
Voice recognition and
translation program
translated speech in
English given by
Richard Rashid,
Microsoft’s top scientist,
into Mandarin Chinese.
https://www.youtube.com/watch?v=xpoFSoTnBpU&t=911s
6. Disrupting The Speech Industry: Going Deeper
After little improvement for 10+ years by the research community…
…MSR reduced error from ~23% to <13% (and under 7% for Rick Rashid’s S2S demo in 2012)
7. Disrupting The Speech Research in Academia
“This joint paper from the major speech recognition laboratories
was the first major industrial application of deep learning.”
9. Components of Speech Recognition System
Separate Speech Recognition Models Unified by End2End Deep Learning
Training Data
Applying Constraints
Search
Recognized Words
Representation
Speech Signal
Acoustic Models Language ModelsLexical Models
17. Three Challenges Unique To Investment Management
1
Very low signal-to-noise
ratio
2
Strong nonstationarity
with adversarial nature
3
Heterogeneity of big
(alternative) data
18. Three Challenges Unique To Investment Management
1. Very low signal-to-noise ratio
The technology used to combat noise shares
characteristics with the technology used to handle
small data in training large AI systems, including:
Ability to exploit structure in data
Reliance on prior knowledge
Use of data simulation/augmentation
Smart model regularization
Etc.
AI problems outside finance generally have
lower noise levels, for example:
Speech
Machine translation
Language understanding
Image/video classification & detection
Medical diagnosis
19. Three Challenges Unique To Investment Management
2. Strong non-stationarity with adversarial nature
20. Three Challenges Unique To Investment Management
2. Strong non-stationarity with adversarial nature
Contrast: nonstationary signals
with no adversarial nature
22. Additional Constraints Applying To AI In Investment Management
What still needs to be done to ensure success?
Data
Access
Respect
for Privacy
Scarcity
of Talent
Tailored
Algorithms
23. This document and the information it contains is strictly confidential
and may not be disclosed to any persons other than those for whom it
is intended, nor should this document or the information it contains be
copied, distributed, or redistributed, in whole or in part, without the
prior written consent of Citadel.
All trademarks, service marks and logos used in this document are
trademarks or service marks or registered trademarks or service marks
of Citadel.
Thank You !
Notas do Editor
Hi, I’m LI Deng, Chief AI officer of Citadel, a
Specialist in AI and machine learning, information theory and statistics, speech, NLP, and now finance.
Before I start, I would like to thank YYY/ZZZ for inviting me here to share my and company’s perspectives on AI in Finance.
In the remaining time, I would like to cover three closely related topics:
First, will AI transform the financial markets? The answer is of course positive (otherwise I would not accept to speak here on the topic of AI in Finance). But I would like to provide some rationale behind the answer, from the perspective of high successful AI (deep learning) in other industries which I had first-hand experience in my past career.
Transitioning from speech/NLP to finance industries, in term of the past, present and future of AI, leads to the second topic of Technical Challenges that are unique to the finance industry.
This is followed by the third topic of other less technical constraints and challenges in applying AI to finance, investment in particular.
Modern AI or deep learning is advanced technology increasingly relevant to finance.
As I am sure all of us in the audience know, with MIT Tech Review credited for rapid dissemination of such progresses, deep learning has disrupted speech, vision, NLP, and robotics industries over the past decade.
Due to the time limit, I will have time only to focus on speech industry here and in next few slides.
The connection between speech and finance industries, on surface, seems obvious.
- For one thing, since 90’s, a number of speech recognition experts specialized in shallow statistical machine learning (e.g. HMMs) have moved away from speech industry to become well known leaders in hedge fund industry.
More technically, both speech signals and the financial market data are in a similar form of non-stationary time series, from which deeper information is extracted for the purpose of predicting linguistic symbols or of forecasting future stock values.
However, beyond these superficial similarities, much larger technical differences stand out between finance industry (which has not yet been significantly impacted by AI) and speech/vision/NLP (which are towards maturing due to deep learning).
Cut below
=========================================
[Li will change this slide to remove CV & NLP; not enough time]
AI and deep learning is advanced technology increasingly relevant to finance
provide the potential to unlock large positive benefits for society
Immense amounts of data and resources available in the financial industry and capital markets
big data essential for AI and deep learning
unlike other industries, most financial/market data public or easily obtainable
despite data availability, not all of it is being used or used to max effectiveness
Deep learning and AI piece together massive, diverse data sets
in ways that they can be beneficially incorporated into financial markets
big data ((un)supervised): hallmark of deep learning and modern AI
Let me briefly reflect on the path of modern AI (deep learning) in disrupting speech recognition industry, giving rise to today’s prominent products of Microsoft’s Cortana, Amazon’s Alexa/Echo, Google Assistant, and Apple’s Siri.
After many years of slow progress in speech recognition using (shallow) machine learning (HMM-GMMs), the launch of deep learning into this 40 year-old field started at NIPS-2009.
I was fortunate to co-organize this event with Prof. Geoff Hinton (my consultant), and with his graduate students (my interns) working closely with my speech recognition group at Microsoft Research in Redmond in coming 1.5 years.
[two students, one at Microsoft and Amazon leaving the same day as I; another at Google Brain after turning down my best offer at MSR contributing to the “high pay” of deep learning fresh Ph.D. in media]
We at Microsoft took then academic idea of deep learning with promising results in a very small phone recognition task to several stages of increasingly larger industry scales of very large vocabulary conversational speech recognition.
Cut below
==============================
Invitee 1: give me one week to decide …,…
Not worth my time to fly to Vancouver for this…
Invitee 2: A crazy idea… Waveforms for ASR are not like pixels for image recognition. It is more like using photons!!!
After two+ years of intense work at Microsoft, with Hinton twice visiting Redmond in 2009 and 2010 working side-by-side with me on deep learning, and with two of his students interning with me, speech recognition error rate was cut by about half.
Then, in the fall of 2012, a public demo in China was carried out, with 3000 people in the audience, voice recognition and translation program successfully translated speech in English given by Richard Rashid, into Mandarin Chinese with virtually no error.
This impressive event was reported in this NY Times’ full-page article (John Markoff interviewed me at Microsoft). Words quickly spread out about this very first industry-scale success of deep learning.
Now let me return to the connections between speech models to finance models.
The main success of deep learning during 2009-2011, attributed to the collaborations between Microsoft and U. Toronto, lies in the use of DNNs to unify several (but not all) major components in the full modeling and recognition process. (Andrew Ng later in Baidu unified all components, and threw away lexical models).
We believe this type of success in speech may inspire future successes of deep learning in finance, at least at a high, strategic level.
My learning process (from Microsoft AI, worked on speech, language, business process, marketing/sales, not finance), as a “grad student”, to learn from Citadel’s superstar finance experts and from books. This is a quite insightful book, fitting my level (a few months ago) well, with “black box” in the title that describes trading process much like we describe deep leaning models.
Modular structure:
Analogous to speech recognition modules: feature extraction, acoustic models such as HMM, language model, pronunciation model.
All feeding into a module that optimize components
Then “execution model” is like decoder in online deployment
Speech recognition revolution: from HMM-centric modular structure and modular learning to holistic DNN structure and end2end learning
My work with Geoff Hinton & students (2009-2010) integrated feature extraction and HMM (dropping errors by 40% with 1.5 years of work), bringing in deep learning to industry from academia
Subsequently, Andrew Ng (Baidu), Google etc. integrated LM and Pron models as well (truly end2end), reducing errors further ~20%.
Amazon, Apple also did deep learning later (not published), so we have Alexa, Siri speech, in addition to Microosft Cortana, Google Assistant.
Now you may guess whether the same concept of holistic modeling and E2E learning may apply to finance (and how), depicted in this box.
However, as alluded earlier, drastic technical differences stand out between finance and speech/vision/NLP. Let me now address three significant challenges that are unique to financial investment management, one by one.
AI problems outside finance generally have lower noise levels
Examples: speech, machine translation, language understanding image/video detection and prediction, medical diagnosis
The technology used to combat noise shares characteristics with the technology used to handle small data in training large AI systems, including:
ability to exploit structure in data,
reliance on prior knowledge,
use of data simulation/augmentation,
smart model regularization, etc.
The adversarial nature in financial market is very different from that in other applications such as playing board-games and fighting robots
Financial academia have yet to propose an effective model for addressing nonstationarity in financial markets due to adversarial competition
Recent literature in AI and robotics is shedding some light - recent papers by Google-DeepMind, Berkeley and Open AI
There has been an immense proliferation of data in recent years: Market price data, fundamental data, and huge sets of “alternative” data including text, image, voice, and multimedia
JP Morgan recently compiled comprehensive sources of such alternative data useful for forecasting financial market
Many other useful data are proprietary to private individuals and data owners (e.g. click streams from search engines, user data from cell phone usages, product ordering from Alexa at home, etc.)
Access to data is incomplete given that a lot of information is proprietary or private
Need more sharing of data for training deep learning models, as long as this does not violate fiduciary duties of finance firms to their clients and it does not harm privacy of individuals
Scarcity of talent capable of bridging the gap between AI research and finance (need to cultivate a talent base that understands both technology and the financial markets)
Need for advanced algorithm development tailored to the financial market