The document describes the process of developing and productionizing a recommendation engine for BBC Sounds. It discusses: 1) The initial challenge of replacing an outsourced recommendation engine and prototyping a new one using factorization machines. Qualitative user tests showed improved recommendations over the external provider. 2) Productionizing involved using Google Cloud Platform, Apache Airflow for workflows, Apache Beam for efficient data processing, and precomputing recommendations to serve 1500 requests/second with low latency. 3) Initial A/B tests found a 59% increase in interactions and 103% increase for under 35s using the new recommendation engine. Ongoing work includes optimizing costs and API performance.

1. From an idea to production
Recommender for BBC Sounds
Tatiana Al-Chueyr
Principal Data Engineer at Datalab
MLOps London, 28 September 2021 @tati_alchueyr

2. about you

3. @tati_alchueyr
MLOps London 28 September 2021
3
https://ahaslides.com/MLOPS1

4. @tati_alchueyr
At what stage of MLOps adoption is your team?
4

5. @tati_alchueyr
What do you expect from this presentation?
5

7. @tati_alchueyr
BBC.
Vision
For the BBC to be a leader in Machine Learning that
delights audiences and prioritises the needs of
individuals and society over corporations and states.
Mission
To develop and deploy Machine Learning at BBC scale
so that teams can tailor services to individuals whilst
upholding our editorial values.
7

8. @tati_alchueyr
BBC. . .hummingbirds
8
The knowledge in this presentation is the result of lots of teamwork
within one squad of a larger team and even broader organisation
current squad team members
previous squad team members
Darren
Mundy
David
Hollands
Richard
Bownes
Marc
Oppenheimer
Bettina
Hermant
Tatiana
Al-Chueyr
Jana
Eggink

9. recommendation engine
the challenge

10. @tati_alchueyr
The BBC outsourced a recommendation engine
10

11. @tati_alchueyr
The audience liked personalised recommendations
11

12. @tati_alchueyr
Could we replace it with our own recommender?
12

13. @tati_alchueyr
Could we replace it with our own recommender?
13

14. recommendation engine
algorithms

15. @tati_alchueyr
Content-based approach
15

16. @tati_alchueyr
Collaborative-filtering approach
16

17. @tati_alchueyr
Hybrid approach e.g. Factorisation Machine Algorithm
17

18. @tati_alchueyr
Machine learning workflow
18

19. recommendation engine
the prototype

20. @tati_alchueyr
1-2 months of work:
● Collected data (quick-and-dirty™ scripts)
● Compared existing Python Factorisation Machines libraries (winner: LightFM)
● Trained and predicted recommendations (quick-and-dirty™ scripts)
● Implemented a qualitative experiment tool
● Recruited volunteers to join the qualitative experiment
● Ran qualitative experiment, comparing:
○ External provider recommendations
○ Our own Factorization Machines-powered recommendations
The prototype
20

21. @tati_alchueyr
Qualitative experiment: how
Who
● ~30 test users recruited
○ Internal BBC employees
○ Under 35
How
● Two sets with 9 recommendations each:
○ External provider
○ Internal factorisation machines
● Users, without knowing the origin of the recs, had to:
○ choose “the best”, “both”, or “neither”
○ explain why
21

22. @tati_alchueyr
Qualitative experiments
neither external
provider
factorisation
machines
both
22

23. recommendation engine
productionising

24. @tati_alchueyr
Productionising machine learning
Configuration
Data Collection
and
Transformation
Feature Extraction
Data
Verification
Machine
Resource
Management
Serving
Infrastructure
Monitoring
Process Management
Tools
Analysis Tools
ML Code
Image copied from presentation by Googler @mpyeager
24

25. @tati_alchueyr
Tech stack
● Google Cloud Platform
● Python used as our main coding language
○ LightFM library used for the model: very handy dataset class and multi-threading processing
● Apache Airflow (Composer) for the workflows orchestration
● Apache Beam (Dataflow) for the parallelizable data processing
● Redis to store our pre-computed recommendations
● Kubernetes for running the API and CPU-intensive tasks
● Terraform for the infrastructure as code development
25

26. @tati_alchueyr
Machine learning workflow
Input
Processing
Output
User activity data Content metadata
Recommendations
Machine Learning model
training
Predict recommendations
26

27. @tati_alchueyr
Machine learning workflow
Input
Processing
Output
User activity data Content metadata
Business Rules, part I - Non-personalised
- Recency
- Availability
- Excluded Masterbrands
- Excluded genres
Business Rules, part II - Personalised
- Already seen items
- Local radio (if not consumed previously)
- Specific language (if not consumed previously)
- Episode picking from a series
- Diversification (1 episode per brand/series)
Recommendations
Machine Learning model
training
Predict recommendations
27

28. @tati_alchueyr
Steps to be done in the workflows, before the API
Input
Processing
Output
User activity data Content metadata
Business Rules, part I - Non-personalised
- Recency
- Availability
- Excluded Masterbrands
- Excluded genres
Business Rules, part II - Personalised
- Already seen items
- Local radio (if not consumed previously)
- Specific language (if not consumed previously)
- Episode picking from a series
- Diversification (1 episode per brand/series)
Recommendations
Machine Learning model
training
Predict recommendations
28

29. @tati_alchueyr
model
Recommendation API strategies
API
API
user
activity
content
metadata
cached
recs
A. On the fly
B. Precompute
predicts & applies rules
retrieves pre-computed recommendations
29

30. @tati_alchueyr
model
Recommendation API strategies
API
API
user
activity
content
metadata
cached
recs
A. On the fly
B. Precompute
predicts & applies rules
retrieves pre-computed recommendations
Goal:
1500 requests/s with
P95 responses < 60
ms
30

31. @tati_alchueyr
Recommendation API: load performance
On the fly Precomputed Precomputed
Concurrent load tests
requests/s
50 50 1500
Success percentage 63.88% 100% 100%
Latency of p50 (success) 323.78 ms 1.68 ms 4.75 ms
Latency of p95 (success) 939.28 ms 3.21 ms 57.53 ms
Latency of p99 (success) 979.24 ms 4.51 ms 97.49 ms
Maximum successful
requests per second
23 50 1500
Goal:
1500 requests/s with
P95 responses < 60
ms
Machine type: c2-standard-8, Python 3.7, Sanic workers: 7, Prediction threads: 1, vCPU cores: 7, Memory: 15 Gi, Deployment Replicas: 1
31

32. @tati_alchueyr
model
Strategies to serve recommendations
API
API
user
activity
content
metadata
cached
recs
A. On the fly
B. Precompute
predicts & applies rules
retrieves pre-computed recommendations
32

33. @tati_alchueyr
Steps to be done in the workflows, before the API
Input
Processing
Output
User activity data Content metadata
Business Rules, part I - Non-personalised
- Recency
- Availability
- Excluded Masterbrands
- Excluded genres
Business Rules, part II - Personalised
- Already seen items
- Local radio (if not consumed previously)
- Specific language (if not consumed previously)
- Episode picking from a series
- Diversification (1 episode per brand/series)
Precomputed
recommendations
Machine Learning model
training
Predict recommendations
33

34. recommendation engine
workflows orchestration

35. @tati_alchueyr
Workflows orchestration: requirements
● Scheduling recurrent jobs
● Retry executing a task if it fails
● Task dependency management
● Monitoring and logs
● Capability of programmatically defining workflows (direct acyclic graphs)
● Built-in support for writing automated tests
35

36. @tati_alchueyr
Workflows orchestration: Apache Airflow
36

37. @tati_alchueyr
Google Managed Apache Airflow: Cloud Composer
37

38. @tati_alchueyr
Cloud Composer: monitoring
38

39. @tati_alchueyr
Limitation of Apache Airflow
● Good for orchestrating tasks
● Not good for processing data within an Airflow worker
○ Separation of concern with runtime
39

40. @tati_alchueyr
Limitation of Apache Airflow
40

41. @tati_alchueyr
Limitation of Apache Airflow
41

42. @tati_alchueyr
Limitation of Apache Airflow
Issue:
Depending on the
volumes of data, a single
PythonOperator task
which usually takes
10 min could take almost
3h!
Consequences:
Overall delay
Blocked worker
42

43. @tati_alchueyr
Limitation of Apache Airflow
Time estimations (in seconds) to predict recommendations using a c2-standard-30 instance (30 vCPU and 120 GB RAM)
43

44. @tati_alchueyr
Limitation of Apache Airflow
Time estimations (in seconds) to predict recommendations using a c2-standard-30 instance (30 vCPU and 120 GB RAM)
2h to predict
recommendations for
10k users
What about 5 million
users - or more?
44

45. @tati_alchueyr
Limitation of Apache Airflow: solutions
Delegating processing to other services
● Tasks which scale vertically (better hardware)
○ Airflow Compute Engine (Virtual Machine) Operator (GceInstanceStartOperator)
○ Airflow Kubernetes Pod Operator (GKEPodOperator)
● Tasks which scale horizontally (can be split and distributed in multiple nodes)
○ Airflow Dataflow Operator (Google Dataflow, Apache Beam )
○ Airflow Dataproc Operator (Google Dataproc, Apache Spark & Hadoop)
45

46. recommendation engine
efficient data processing

47. @tati_alchueyr
Apache Beam
“Apache Beam is a unified
programming model designed
to provide efficient and
portable data processing
pipelines”
47

48. @tati_alchueyr
Apache Beam
https://towardsdatascience.com/running-an-apache-beam-data-pipeline-on-azure-databricks-c09e521d8fc3
48

49. @tati_alchueyr
Apache Beam: overview of Dataflow job
Image from the book “Google Cloud Platform In Action” by JJ Geewax, Chapter 20
49

50. @tati_alchueyr
Apache Beam: overview of Dataflow job
Parallel processing “effortlessly”
Image from the book “Google Cloud Platform In Action” by JJ Geewax, Chapter 20
50

51. @tati_alchueyr
Simple Beam example
https://beam.apache.org/documentation/transforms/python/aggregation/cogroupbykey/
51

52. @tati_alchueyr
Apache Beam: overview of Dataflow job
52

53. @tati_alchueyr
Adoption of Apache Beam & Dataflow
“Serverless” parallel processing of 41,258,135 items (27.32 GB) with
Python in 1min 24s using 10 default workers
53

54. @tati_alchueyr
Pure Airflow
PythonOperator in
Cloud Composer
DataflowOperator
running a Beam
pipeline within
Dataflow
episode
availability episode
s/PythonOperator/DataflowOperator
Computation time reduced almost by one
order of magnitude
Document
type
PythonOperator DataflowOperator Performance
gain
episode 60 min 6 min 90%
availability
episode
12 min 5 min 58%
54

55. @tati_alchueyr
Precomputing recs for millions of users
55

56. @tati_alchueyr
Cost analysis
https://cloud.google.com/dataflow/pricing
Resources metrics per job
56

57. @tati_alchueyr
Cost reduction
300$ per run
57

58. @tati_alchueyr
Cost reduction
memory intensive
transformation
Solutions
● Use shared memory
● Split pipelines so only the memory
intensive transformation uses expensive
machine types.
58

59. @tati_alchueyr
Cost reduction
memory intensive
transformation
Solutions
● Use shared memory
● Split pipelines so only the memory
intensive transformation uses expensive
machine types.
59

60. Overall
Architecture

61. @tati_alchueyr
Overall architecture
61
Availability, metadata and
episode data to support a
snapshot view of sounds
content.
DATA
SOURCE
S
Sounds User activity data for
content consumption. Signed in
only
DATA
PLATFORM
Dedicated stream of UAS
(encrypted) into a Sounds Data
Lake
ML PLATFORM
Dedicated Sounds Metadata
Snapshot built intra-day
User Activity features processed
intra-day
Both sources merged into a feature
set to support the re-training of the
Sounds recommender
RECOMMENDER
Serving layer that exposes the
recommender to Unirecs.
Unirecs serves
recommendations to end
users
Build Cluster for intra-day
build/training of the model.
Dedicated service for
pre-computing
recommendations

62. Early
A/B Tests results

63. @tati_alchueyr
● +59% increase in interactions in Recommended for You rail
● +103% increase in interactions for under 35s
Initial A/B Tests results
63

64. @tati_alchueyr
API Latency
64
https://dashboard.sre.tools.bbc.co.uk/d/-RO7WwbZk/unirecs-live?viewPanel=63&orgId=6&from=now-7d&to=now&refresh=1m

65. ML Ops pyramid choice

66. @tati_alchueyr
Choosing the level of abstraction
66
None
Cloud built-in
proprietary tools
Open source / Proprietary
ready-to-use tools (eg. tfx)
Build custom workflows and train your own
models using Keras, Sklearn, PyTorch,
others
Customisation x Easiness of adoption

67. http://datalab.rocks
67

68. @tati_alchueyr
Thank you very much!