Presentation about the state of AI, policy-relevant AI research and evidence gaps that can be addressed with new data, methods and modelling approaches.

1. Some new directions in the
economics of AI
Juan Mateos-Garcia
16 March 2022

2. About today
Goal
Identify some avenues of research that might yield evidence to inform AI policy.
Hopefully we can pursue some of them together!
Structure
1. Problem definition
2. State of the world
3. State of knowledge
4. New directions

3. 1. Problem definition
Artificial Intelligence
“The designing and building of intelligent agents that receive percepts from the environment and take
actions that affect that environment.”
AI systems are deployed in a way that enhances societal wellbeing
Technology development Economic organisation
Values Policy
?

4. 1. Problem definition: A question of control
S1d
S1a
δ1
S2d
S2a
δ2
P = f(δ,K1
)
t
Policies P informed by our understanding of the state
of the world δ(S1d
,S1a
) and our state of knowledge K
about what to do.
Question: Is KAI
==KIT
?
K1
K2

5. 2. State of the world: Technology development
● We are moving into a
large model era
● Model performance
scales with size… but
not for all tasks
● Important
breakthroughs in AI
for scientific R&D
Sevilla et al 2022

6. 2. State of the world: Economic organisation
● Industrialisation of AI: private sector
generating the most significant
advances
● Emergence of the foundation model
pipeline / business model
● Geopolitical fragmentation of AI
R&D
Ganguli et al 2022

7. 2. State of the world: Values
● LLMs create
significant ethical
risks
● This has a
counterfactual
aspect: values
missing from paths
not explored
● Proliferation of
ethical principles and
guidelines
Weidinger et al 2022

8. 2. State of the world: Policy
● Continued public investment in
AI R&D
● Increased regulatory pressure on
risky AI systems (EU AI act)
○ Including bans e.g. facial
recognition
● Increased attention being given
to implementation and
experimentation

9. 2. State of the world: Summary
AI is evolving towards larger models developed by a small
number of mostly private actors with access to big data and
compute.
Model performance improves with size in many but not all
dimensions, and their deployment raises important ethical
risks.
Policymakers are becoming more assertive about acceptable
/ unacceptable AI applications while continuing to support its
development.

10. 2. State of the world: Underlying patterns
■ Open science / open source model
■ Metric driven innovation speeds pace but induces racing &
gaming
■ Hyper-accelerated technical evolution
■ Architectural / systemic innovation
○ Combinatorial explosion of implementation possibilities: more directionality
■ Hidden and surprising failure modes
■ Highly scalable in a narrow but profitable set of use-cases

11. 3. State of knowledge: The AI “production function”
We can use this to tell stories: “Corporate research teams in the USA and China have leveraged big
private and open datasets and compute to build large language models that are now powering internet
services creating big economic impacts and raising ethical risks”
But we don’t really understand how this new, complex, fast-changing system works
Funding
Research
Code / tech
Models
Data
Talent
Data
Organisations, institutions, ecosystems, polities
Impacts
Compute
Products
Services
Platforms
Companies
Research Code / tech
Models
Data

12. 3. State of knowledge: Some gaps
Funding
Research
Code / tech
Models
Data
Talent
Data
Organisations, institutions, ecosystems, polities
Impacts
Compute
Products
Services
Platforms
Companies
Research Code / tech
Models
Data
How does this open (but uneven)
infrastructure shape development and
diffusion?
What is the link between
diversity in talent and
diversity in ideas
pursued?
Does private sector leadership skew the
technologies that are developed and their
impacts? What about regional and
national angles?
What are the hidden impacts
of the technologies that are
developed and how do they
compare with counterfactual
paths?

13. Economics of AI
(mainstream)
AI governance
AI progress
forecasting
AI metascience
Critical studies
of AI
AI safety
AI ethics
Jobs, income share and
productivity
Inputs and outputs
Scientometrics
Technical failure
modes and risks
Institutions
Power and
justice
Principles and
guidelines
This is partly a consequence of
knowledge fragmentation
Could a complex economics of AI
help to bridge some of these gaps?

14. Constructionist
Quantitative / multi-site
Economics of AI
(mainstream)
Critical studies of
AI
AI progress
forecasting
AI safety AI governance
Deterministic
AI
metascience
Complex
economics of AI
AI ethics
One technological trajectory; AI will be adopted and the role of policy is to accelerate / prepare / adapt
Multiple technological trajectories; AI might not be adopted. The role of policy is to steer / balance and even block
Quantitative / single-site

15. 3. State of knowledge: Some comments on the Econ of AI
■ Model AI progress as a scalar
○ Looking at patents, survey data, some online job ads
■ Strong focus on automation
○ So far ignoring one of the main “exposed” labour markets: Scientific
R&D
■ Some attention to complementarities in organisation, less so in
production
■ Technologically deterministic
○ With some exceptions: Acemoglu and Korinek

16. 3. State of knowledge: Knowledge flows into CEoAI
Economics of AI (mainstream)
AI governance
AI progress forecasting
AI metascience
Critical studies of AI
AI safety
AI ethics
Considering incentives, organisation, economic impacts
Taking into account all inputs / outputs into the AI production function
Paying special attention to R&D dynamics & the impact of AI on science
Not assuming that “AI will work”
Modelling the relationship between organisations and institutions
Acknowledging a broad range of values
Studying the social and cultural determinants of AI’s trajectory

17. 3. State of knowledge: CEoAI requires…
New data sources capturing
important dimensions of AI R&D
New analytics to find AI and
measure its ecosystems,
composition and trajectory

18. 4. New directions: Some examples (my work)
A Narrowing of AI
research (2020)
The Privatisation of AI
research(ers) (2021)
Deep Learning, Deep
Change? (2018)
Gender diversity in
AI research (2019)
Funding
Research
Code / tech
Models
Data
Talent
Data
Organisations, institutions, ecosystems, polities
Impacts
Compute
Products
Services
Platforms
Companies
Research Code / tech
Models
Data
AI and the fight against
Covid-19 (2020)
Mapping
innovation
missions
(2019)

19. 4. New directions: A Narrowing of AI research?
Is AI research becoming thematically
narrower?
■ Analysis of AI preprint corpus enriched
with institutional information
■ Use of topic modelling to characterise
diversity
■ Evidence of stagnation in thematic
diversity
■ Private companies less diverse (and
more influential)

20. 4. New directions: The privatisation of AI research(ers)
Is there a brain drain of AI researchers into
industry? What are its drivers and impacts?
■ Analysis of AI articles enriched with
institutional information
■ Survival analysis of researcher transition
from academia to industry
■ Industry hiring influential male
researchers with expertise in deep
learning
■ Ambivalent link between transitioning
and influence: short-term benefits offset
by decline in citation levels over time

21. 4. New directions: More examples
Factors driving advances in AI
benchmarks (Martinez-Plumed
et al, 2021)
Evolution in the use of data in ML research
(Koch et al, 2021)
Impact of public
funding on AI’s
trajectory (Iori et al,
2021)
Interactions between national and corporate
innovation system (Åke-Lundvall and Rik,2022)
Funding
Research
Code / tech
Models
Data
Talent
Research Code / tech
Data
Organisations, institutions, ecosystems, polities
Impacts
Compute
Models
Products
Processes
Platforms
Companies
Data
Interaction between AI
& hardware (Hooker,
2020, Pryktova et al,
2021)
Organisational
implications of AI
(Bresnahan, 2019)
Historical role of
power in shaping AI’s
trajectory (Mohamed
et al 2020))

22. 4. New directions: Future possibilities [code]
What can we learn about AI’s structure and
trajectory from its open code ecosystem?
■ Papers with Code matches AI research with
benchmark performance and open source code
■ GitHub has an open API to collect data in real time
Example questions:
■ Why are private companies so active in the open
source space?
■ Is research fragmenting along the lines of open
source frameworks?
■ Can we use open source data to measure diffusion?

23. Could we use GitHub data to capture the network structure of AI software and its evolution?
Valverde and Solé (2015)

24. 4. New directions: Future possibilities [Trajectories]
What institutional factors are narrowing AI’s
trajectory and what are its impacts?
■ We know what papers have been accepted and
rejected in prestigious conferences like NeurIPS.
■ We know the tasks (benchmarks) that are tackled by
different papers (from PwC)
Example questions
■ Is the review process narrowing AI’s trajectory?
What about publication races?
■ What is the value of non-mainstream trajectories in
e.g. tackling different problems / injecting novelty
into the mainstream?

25. 4. New directions: Future possibilities [Impacts]
Do technical limitations skew AI applications
towards less societally beneficial areas?
■ We can use research / funding metadata and
semantic analysis to study where AI is being
applied (and by whom) in specific domains e.g.
health
Example questions
■ Are AI applications skewed towards data-rich
domains?
■ What is the role of non-mainstream techniques in
diversifying AI’s focus?
■ Who generates / funds what applications?

26. Who is funding AI projects to tackle chronic diseases with different topic compositions?
Mateos-Garcia (2019)

27. 4. New directions: Future possibilities [Impacts]
What is the impact of AI in scientific labour markets?
■ Deep learning is set out to transform / already
transforming scientific R&D in areas such as structural
biology, genomics, materials science.
■ Availability of open source software is playing an
important role
Example questions
■ Can we use scientometric methods to build skills maps
in research fields and quantify their exposure to AI?
■ How are researchers moving through these maps in
response to exposure? What are the risks?

28. Can we build maps like this about exposure to automation in e.g. structural biology?
Sleeman et al (2020))

29. Conclusions
■ There are big gaps in our understanding of AI’s trajectory that could hinder
policy
■ These are linked to disciplinary silos. In particular, mainstream economics of
AI is neglecting some important questions in the field.
■ A complex economics of AI could help bridge this gap
■ This will require working with new data sources and (data science) methods
■ There are many interesting questions that we could explore
■ We need institutional innovations to produce timely, policy-relevant evidence
in this fast-moving domain 🏃🏃🏃🏃🏃

30. Thank you.
Questions?
juan.mateos-garcia@nesta.org.uk
@JMateosGarcia