Responsible machine learning at the BBC

Ethical machine learning
building recommendation engines with editorial support
Tatiana Al-Chueyr
Senior Data Engineer
London, 9 July 2020 @tati_alchueyr

@tati_alchueyr
About me
● Brazilian living in London since 2014
● Senior Data Engineer at the BBC Datalab team
● Graduated in Computer Engineering at Unicamp
● Passionate software developer for 16 years
● Experience in the private and public sectors
● Developed software for Medicine, Media and Education

@tati_alchueyr
Campus Party & I
● First time “campuseira” at Campus Party Brasil 2008
● Invitation from “uncle” & organiser: Walmir Cardoso
● Lots of fun, camping and gave a talk
● This is my fifth Campus Party - the first fully digital!

@tati_alchueyr
BBC
● British Broadcasting Corporation
● Values
○ Independent, impartial and honest
○ Audiences are at the heart of everything we do
○ We take pride in delivering quality and value for
money
○ Creativity is the lifeblood of our organisation
○ We respect each other and celebrate our
diversity so that everyone can give their best

@tati_alchueyr
BBC
● Founded in 1922
● Purpose
○ Inform
○ Educate
○ Entertain
● “Our organisation exists in order to serve individuals and society as a
whole rather than a small set of stakeholders.”
Reference: Gabriel Straub (BBC)

@tati_alchueyr
bbc.stats()
● BBC TV reaches 91% UK adult population
● BBC News reaches 426 million global audience weekly
● ~2000 pieces of BBC content are produced every day ….
and a limited number of available slots to occupy!
Reference 1: BBC
Reference 2: BBC
Image Credit: BBC

@tati_alchueyr
BBC. .
“Bring the BBC’s data together
accessible through a common platform,
along with flexible and scalable tools to
support machine learning to enable
content enrichment and deeper
personalisation”

@tati_alchueyr
BBC. .
Mission
To develop and deployMachine Learning at BBCscale so that teams cantailor services to individualswhilst upholding our editorialvalues.
Vision
For the BBC to be a leader in Machine
Learning that delights audiences and
prioritises the needs of individuals and
society over corporations and states.

@tati_alchueyr
Pre-lockdown Datalab team members (15 August 2019)
BBC. .

@tati_alchueyr
Locked-down Datalab team members (19 March 2020)
BBC. .
COVID-19
pandemic

@tati_alchueyr
BBC. .
● Multi-disciplinary team
○ Architecture
○ Data science
○ Editorial
○ Engineering
○ Product Management
○ Project Management
@tati_alchueyr

@tati_alchueyr
BBC Machine learning applied to the audiences
Image credit: BBC

@tati_alchueyr
BBC Machine learning applied to content creation
Image credit: BBC
Made by the Machine: when AI met the archive (BBC 4)

@tati_alchueyr
Machine learning overview

@tati_alchueyr
Our current work
● We delivered a new recommendation engine for
World Service News in Arabic and Hindi. Russian
is also ready (waiting for UX changes)
● We have developed recommendation engines for
BBC News and BBC Sport

@tati_alchueyr
Our current work
● We are close to shipping a recommendation engine
for Sounds so that we can replace the existing
provider on the recommendation rail
● We are working with the Voice team as well to help
personalise the experience
● We are also exploring how to best help iPlayer

@tati_alchueyr
How do our recommendations engines work?

@tati_alchueyr
Our typical workflow

@tati_alchueyr
Qualitative Experiment
Who
● ~30 test users recruited
○ From non-editorial and editorial teams from BBC audio networks
○ Under 35
How
● Two sets of recommendations displayed
● Users have to pick either the best list, or “both”, or “neither”
● And explain why

@tati_alchueyr
Qualitative Experiment Feedback
● “Need to categorize speech vs music,
background listening vs ‘serious’ content”
● “Need to consider the age of the item”
● “Looking for diverse content durations …”
Reducing item/user biases helped to generate
more personalised recommendations than the
current state
Neither Content-
Based
Hybrid
approach
Both
2 8 17 1
7% 28.5% 61% 3.5%

@tati_alchueyr
Quantitative Experiment (MVT or A/B test)

experimental personalised app
+

@tati_alchueyr
BBC+ app experiment
● Fully personalised experience on short videos, on Android & iPhone
● Allow users to find gems that they didn’t know at a time that suits them

@tati_alchueyr
BBC+ app experiment
● How to get from algorithm to product
○ Start with content-based recommendations
○ Apply business rules

@tati_alchueyr
Legal, editorial, GDPR, business values
https://www.bbc.com/editorialguidelines/

@tati_alchueyr
Legal Policies
Programme: BBC
Contempt of court
● The recommendations should not affect the
outcome of a legal case
● The BBC can be held accountable for influencing
the jury’s opinion
Action
● Create a “contempt of court risk” label by detecting keywords
such as arrest, assault, allegation etc
● Avoid items with this label

@tati_alchueyr
Legal Policies
Electoral law
● During elections we should not surface
political content that could influence the vote
Action
● Create a “political risk” label by detecting
political content sources
● Avoid items when appropriate

@tati_alchueyr
Editorial Policies
Quality criteria
● Avoid content that shows little care has been
taken in the metadata
Action
● Avoid content with poor titles and
descriptions

@tati_alchueyr
Editorial Policies
Under 16 audience
● Provide children-safe content
● BBC’s 9PM watershed
Action
● Avoid items with warnings of sex, violence,
strong language

@tati_alchueyr
Cold start: human curation alongside automation

@tati_alchueyr
GDPR
Explainability
● Choose simple models over complex ones
● UI features to provide explanations
Agency
● UI features for users to interact with the algorithm
● Eg. delete history items, like, dislike, report

@tati_alchueyr
Curation values
● Affection
● Authenticity
● Compelling
● Fresh
● Warm
● Quirky
● Relatable
● Aspirational
● Entertaining
● Reassuring
Reference: Anna McGovern
“Recommendations Editorial Lead” at the
BBC
Much more than click rates

@tati_alchueyr
Business values & objectives
Quantitative offline evaluation
● NDCG, hit rate, diversity, recency, surprisal
● Prioritise diversity and recency over accuracy
Qualitative offline evaluation
● Prioritise content for young audiences
● Prioritise content of editorial importance

@tati_alchueyr
BBC+ app experiment

@tati_alchueyr
BBC+ app experiment
Takeaways
● The editorial partnership is key to how we work
● The company’s principles are at the heart of all of our decisions
● There is a significant path between implementation and production ready

@tati_alchueyr
Public Service Role in getting AI done right
● Informing the Debate: given our role as broadcaster, help make sure
there is a truly informed debate
● Bringing Partners Together: public service institutions, academia,
and the commercial sector around the biggest issues, and by sharing
our combined knowledge
● Responsible Technical Development: using responsible machine
learning to enrich users’ lives in a way that upholds our public service
values of impartiality, independence, accountability and universality
Reference

@tati_alchueyr
The BBC Machile Learning Pubic Commitments
1. Audiences at the heart of everything we do. We celebrate diversity
○ Good value for money and focusing on using the audience-based data to improve
their experience
3. Our algorithms serve our audiences equally and fairly, so that the full breadth of the
BBC is available to everyone
5. Where ML engines surface content, outcomes are compliant with the BBC’s editorial
values. We will also seek to broaden, rather than narrow, our audience’s horizons
6. Algorithms form only part of the content discovery process for our audiences, and sit
alongside (human) editorial curation Reference: Gabriel Straub (BBC)

@tati_alchueyr
Machine Learning Principles
● Principles and tools to ensure we avoid common pitfalls around machine learning
● Checklist
○ A list of questions for ML practitioners to work through and review
○ It has been developed by an interdisciplinary group, drawing from best practice
within the BBC and the industry
○ Self-audit tool: it is entirely up to team how they use it.. The intention is not to
create a process for the sake of it - we want to make thinking happen.
MLEP Checklist

@tati_alchueyr
Flourishing in the age of AI
● Research
● 11,000 people
● 7 markets
● What people want from their lives
● How technology might enable that
Reference: Flourishing in AI report

@tati_alchueyr
“(...) people in the UK don’t think technology is being
developed with their best interests at heart”

@tati_alchueyr
● How satisfied are you with
your life?
● To what extent the thing
you do in life is
worthwhile?
● How anxious did you feel
yesterday?
Base: 5432, May 2019

@tati_alchueyr

@tati_alchueyr
How does the BBC personalise
● Commitment to transparency and explainability for audiences
● Explainers of personalisation
○ https://www.bbc.co.uk/usingthebbc/account/how-is-the-bbc-personalised-to-me/
● What does recommended for you mean
○ https://www.bbc.co.uk/usingthebbc/account/what-does-recommended-for-you-mean/

@tati_alchueyr
Ethical Machine Learning
● How do you make decisions about what is fair?
● Which metrics can you use?
● How to achieve an ethical machine learning in your work?
Reference: Avoiding the Fate of Icarus
Medium

STAY H ME
STAY CONNECTED
SAVE LIVES

thank you
obrigada
gracias
dzień kuje
merci
@tati_alchueyr

Responsible machine learning at the BBC

Recomendados

Recomendados

Mais conteúdo relacionado

Semelhante a Responsible machine learning at the BBC

Semelhante a Responsible machine learning at the BBC (20)

Mais de Tatiana Al-Chueyr

Mais de Tatiana Al-Chueyr (20)

Último

Último (20)

Responsible machine learning at the BBC