1.
Deep Learning, Deep Change
Mapping the development of the Artificial Intelligence General
Purpose Technology
Joel Klinger, Juan Mateos-Garcia and Konstantinos Stathoulopoulos
SPRU Friday Seminar
16 November 2018

2.
The news this week: more than Brexit
Artificial Intelligence is a powerful technology with widespread
applicability… but is it a General Purpose Technology?

3.
The global picture
Countries across the world are putting in place national strategies to
develop their AI sectors… but what is its geography?

4.
The local picture
Geographical proximity could help coordinate the development of
complex AI technologies... What local factors matter?

5.
Structure
1. Theory
2. Defining AI
3. Method and data
4. Findings
a. Is AI a GPT?
b. How is it geography changing?
c. What drives these changes?
5. Conclusions and next steps

6.
Theory. General Purpose Technologies as engines of growth
Technologies ‘characterized by the
potential for pervasive use in a wide
range of sectors and by their
technological dynamism’
[Bresnahan and Trajtenberg, 1995]
● Novelty
● Disruption
● Require complementary
investments
● Generate externalities (risk of
coordination failures)

7.
Literature. Geography
GPT → change in drivers of advantage → New
geographies of production and innovation
[Rosenberg and Trajtenberg, 2004]
https://fineartamerica.com/featured/3-
corliss-steam-engine-1876-granger.html
GPT discontinuity → early product life-cycle →
windows of opportunity
Followed by maturity and consolidation.
[Abernathy and Utterback, 1977, Anderson and Tushman 1990, Klepper, 1996,
Scott and Storper 2003]

8.
Literature. Relatedness
National (regional, local) economies
develop by diversifying into related
sectors / disciplines.
[Delgado et al, 2018]
Co-location with relevant sectors
could be important/necessary for
complex GPTs that require
coordination in deployment.
[Balland and Rigby, 2017]
https://phys.org/news/2007-08-nation-position-product-space-
economic.html

9.
Defining AI. History
https://hackernoon.com/ai-in-medicine-a-beginners-guide-a3b34b1dd5d7
A long dream of
machines that ‘think’
And a long journey of disappointing
implementations

10.
Defining AI. A modern approach
http://cdn.aiindex.org/2017-report.pdf
https://www.quora.com/How-do-I-learn-Neural-Network-by-myself
More data More processing power Software innovations
Deep learning

11.
Defining AI. Policy questions
AI is being described as the latest GPT
[Cockburn et al, 2017, Agrawal et al, 2018]
But is it so? Brittle, narrow, opaque. Maybe not
so general.
[Marcus, 2018]
https://www.theatlantic.com/technology/archive/201
8/03/can-you-sue-a-robocar/556007/
http://lifeinvestasset.com/noticias/reporte-lifeinvest-
15-06-18/
A global race in AI is afoot with dozens of
national strategies being launched. But what
is their economic rationale?
[Stix, 2018, Goldfarb et al, 2018]

12.
Method and data. Research questions
GPT
Geographical
change
Driven by presence
of complements
Q1. Does AI behave
like a GPT?
Q2. Is its geography
being transformed?
Q3. What is the role of
regional
complements?

13.
Method and data. Data pipeline
arXIv
CrunchBase
Deep Learning
papers
Filtering
Deep Learning
clusters
Geocoding
Related research
clusters
Related industry
clustersSemantic
proximity to DL
Co-occurrence
with DL
Q2
Q3
Topic
modelling
Q1

14.
Method and data. arXiv
Popular pre-prints website in physics, engineering and computer science.
1.3m papers. Very popular in the AI community

15.
Method and data. arXiv processing
arXIv Deep Learning
papers
Filtering
Deep Learning
clusters
Geocoding
Related research
clusters
Similarity with DL
Topic
modelling
We fuzzy-match
ArXiv with Microsoft
Academic Graph to
retrieve citations
(for QA) and
institutions (90%
match rate)
We fuzzy-match institutions with Global Research Identifier
(GRID) to extract institution locations. We then bin those into
countries / regions using a point on polygon approach and
Natural Earth shapefiles
We use CorEx, a topic modelling algorithm to identify 2
topics related to Deep Learning. CorEX looks for clusters of
words in the data that maximise correlations in the data
We measure cosine distance
between DL and other computer
science sub-disciplines based on
co-occurrence in papers

16.
Method and data. Anchored correlation explanation (CorEX)
Identify topics in
corpora of text. Looks for
topics that maximally
explains dependencies
between words in
documents.
Does not require
selection of k (number
of topics) like LDA.
Can be seeded with
anchor words
[Gallagher et al 2017]

17.
Method and data. Computer science subject proximities with DL
Disciplines such as
computer vision,
learning, machine
learning, neural
nets and AI
appear closer to
DL

18.
Method and data. arXiv content
We end with 130k unique papers and 250k paper-institution pairs. We identify
15k papers as Deep learning.
This is what the data look like

19.
Method and data. CrunchBase
Global startup-up directory with ~ 450,000 entries (257,000 organisations).
Includes sectors and descriptions. Increasingly used in economics and
management research.
[Dalle et al, 2017, Menon et al 2018]
CrunchBase Related industry
clustersSemantic
proximity to DL
Train a machine learning model
on the CrunchBase data and
out-of-sample predict to what
sectors do arXiv papers relate.

20.
Method and data. Sector relatedness with DL
Sectors such as data,
AI, software,
hardware, science
and engineering,
education and ICT
are more
semantically related
to DL

21.
Findings. Q1: Is DL a GPT?
Test 1: Dynamism
DL is rapidly gaining
importance in
absolute and
relative terms in the
arXiv corpus

22.
Findings. Q1: Is DL a GPT?
Test 2: Generality
[We compare pre and post 2012 because
2012 is generally considered a watershed
moment for DL with the publication of
Krizhevsky et al, 2012.]
DL is being
increasingly applied
in more computer
science disciplines

23.
Findings. Q1: Is DL a GPT?
Test 3: spin out impact. DL papers are over-represented in
the high citation groups in most CS sub-disciplines.

24.
Findings. Q2: How is the geography of DL evolving?
National specialisation
[Based on Revealed Comparative Indices, focusing on high activity
places and higher quality papers]
Change is afoot. China, Singapore and Canada ascending, US
keeps up, Europe (excepting Switzerland and UK) fall behind.

25.
Findings. Q2: How is the geography of DL evolving?
Regional picture
We see some leading digital/creative/
defence clusters amongst the most DL
specialised regions.

26.
Findings. Q2: How is the geography of DL evolving?
Volatility and concentration
After an initial period of volatility (fat
tailed distribution of specialisation),
we see an apparent shake-up,
increasing concentration in a small
number of regions (nations).

27.
Findings. Q3: What are the drivers of regional specialisation?
[All totals logged, all variables normalised, focusing on the top
quartile of locations by arXiv activity]
Persistence +
Research specialisation ~
Industry specialisation ~
Complementarities ++
China ++

28.
Findings. Q3: What are the drivers of regional specialisation?
Complementarities are more important for DL / other data CS disciplines.

29.
Conclusions. Wrap up of findings
1. DL looks like a General Purpose Technology
■ Widespread exploration of opportunities
2. Evidence of discontinuity in its geography
■ But things seem to be settling down
3. Presence of relevant industries linked to DL cluster
development.
■ Although also some evidence suggestive of
unexpected spillovers

30.
Conclusions. Policy implications
1. Further evidence of localised knowledge spillovers
from AI research activity: might justify activist DL
policies.
2. But is the window of opportunity for DL closed now?
3. DL spreading widely despite some concerns about its
narrow / brittle nature. What is the role of R&I funders in
diversifying the AI science/technology trajectory?
4. And what about getting DL applied in less related
sectors (in missions)?

31.
Conclusions. Limitations and next steps
1. Data caveats: Triangulate with other data (traditional
publications, patents)
2. Causality:
a. Look for natural experiments / exogenous shocks
b. Explore mechanisms for research - industry
complementarities (networks, labour flows?)
3. Directionality
a. Further analyse text descriptions to understand sub-
branches of modern DL research and their drivers.

32.
Appendix. GitHub code and data
https://github.com/nestauk/arxiv_ai

33.
nesta.org.uk
@nesta_uk
juan.mateos-garcia@nesta.org.uk