Powering machine learning workflows with Apache Airflow and Python

1. Powering
machine learning workflows with
Apache Airflow and Python
@tati_alchueyr
OctopusCon Python Edition
Kharkiv, 16 November 2019

2. мне приятно быть здесь с
тобой
большое Вам спасибо
Image credit: Turkish Airlines

3. tati_alchueyr.__doc__
● Brazilian living in London since 2014
● Senior Data Engineer at the BBC Datalab team
● Graduated in Computer Engineering at Unicamp
● Passionate software developer for 16 years
● Experience in the private and public sectors
● Developed software for Medicine, Media and Education
● Loves Open Source
● Loves Brazilian Jiu Jitsu
● Proud mother of Amanda
Software InVesalius 3

4. Distance from
Rio de Janeiro to London:
9,272 km

5. Distance from
London to Kharkiv: 2,899 km

6. help(bbc)
● British Broadcasting Corporation
● Values
○ Independent, impartial and honest
○ Audiences are at the heart of everything we do
○ We take pride in delivering quality and value for money
○ Creativity is the lifeblood of our organisation
○ We respect each other and celebrate our diversity so
that everyone can give their best
● Purpose
○ Inform
○ Educate
○ Entertain
New Broadcasting House
London, UK

7. bbc.stats()
➢ BBC TV reaches 91% UK adult population
➢ BBC News reaches 426 million global audience weekly
Reference 1: BBC
Reference 2: BBC
Image Credit: BBC

8. bbc. .mission
“Bring the BBC’s data together
accessible through a common platform,
along with flexible and scalable tools to
support machine learning to enable
content enrichment and deeper
personalisation”

9. Some of the Datalab team members (15 August 2019)
bbc. .mission

10. Have you ever
built or used a machine learning
model?
no yes ?

11. machine learning application personalisation
Image credit: BBC

12. machine learning application content creation
Image credit: BBCMade by the Machine: when AI met the archive (BBC 4)

13. machine learning training & prediction supervised

14. machine learning tools jupyter notebooks
Image credit: Jupyter.org
BBC Datalab
ML course

15. machine learning tools python scripts

16. machine learning tools (remote) servers

17. machine learning tools containers
Image credit: RedHat

18. have you ever
scheduled processes?
no yes ?

19. scheduling humans

20. scheduling processes cron jobs
0 2 * * * /bin/sh backup.sh

21. scheduling processes k8s jobs
apiVersion: batch/v1
kind: Job
metadata:
generateName: backup-
namespace: sample
spec:
template:
spec:
containers:
- name: backup
image: alpine:3.9
command: ["sh", "-c", "backup.sh"]
backoffLimit: 3

22. scheduling workflows cron jobs
Several cron jobs running...
It seems a critical job didn’t run
last night...
Didn’t it run? Did it fail?
Why could it have failed?
Original image credit: XKCD

23. scheduling workflows tools
Azkaban
●

24. how much do you know about
Apache Airflow?
none basic mid high

25. airflow
Image credit: House Plans

26. air(bnb)flow
Airflow release blog post by Airbnb
https://github.com/apache/airflow Image credit: Airbnb

27. airflow why
● Handle complex relationships between jobs
● Handle all the jobs centrally with a well defined user
interface
● Error reporting and alerting
● Viewing and analyzing job run times
● Security (protecting credentials of databases)

28. airflow why not
● In many cases, cron jobs are the simplest and most
effective tool
● Airflow is a complex tool made of several components
○ Learning curve
○ Infrastructure management cost

29. airflow concepts (i) DAG
● All workflows are considered to be DAGs
○ DAG: Direct Acyclic Graph
nodes
direct edge

30. DAG
not
DAG
Job A Job B Job C

31. DAG
not
DAG
Job A
Job B
Job C
Job D

32. DAG
not
DAG
Job A
Job B
Job C
Job D

33. DAG
not
DAG
Job A
Job B
Job C
Job D

34. DAG
not
DAG
Job A
Job B
Job C
Job D

35. airflow concepts (ii) DAG properties
● DAGs (usually) have:
○ schedule
○ start time
○ unique name (ID)
○ nodes: jobs (instances of Operators)
○ edges: dependencies between the nodes

36. airflow concepts (iii) operators
● Operators define the task or job
○ BashOperator: execute shell commands/scripts
○ PythonOperator: execute Python code
○ BranchPythonOperator: execute a code if condition
○ SlackOperator
○ (...)
○ Custom operators

37. demo
Apache Airflow pipeline
(example with a Python operator)

38. airflow concepts (iv) relationships
● Edges define dependencies
○ When some tasks need to execute one after another
Image credit: Airbnb

39. airflow concepts (v) connections
● Connections encrypt credentials
○ The jobs do not need to worry about securing
credentials
Image credit: Airbnb

40. airflow concepts (vi) visualisation
Image credit: Airbnb

41. airflow concepts (vi) visualisation
Image credit: Airbnb

42. demo
Apache Airflow pipeline
(example of data ingestion at the BBC)

43. inside airflow

44. airflow architecture

45. airflow managed service GCP Cloud Composer
Image credit: Google

46. scars of experience
Image credit: XKCD

47. scars of experience installing python packages

48. scars of experience installing python packages
● When using a Python Operator, the job is run within the
worker
● Therefore, by default, Python dependencies are
installed globally to the workers
● In other words, application deployments can break your
Airflow environment

49. scars of experience installing python packages
● Isolate the execution from the scheduling, when
reasonable
● To debug native operators means to debug Airflow itself
● Alternatives to isolate them:
○ PythonVirtualenvOperator
○ DockerOperator
○ KubernetesPodOperator
○ GceInstanceStartOperator
Interest reading: Medium

50. scars of experience debugging
There was a breaking change in an
Airflow plugin, the scheduler couldn’t
process the DAG

51. scars of experience debugging
The DAG in the worker instances was
deleted but its metadata was no longer
available in the scheduler

52. scars of experience debugging
● Error messages are not always obvious
○ Understand what is happening in the system
○ The webserver and scheduler are independent
processes

53. scars of experience versioning can be tricky

54. scars of experience versioning can be tricky

55. scars of experience versioning can be tricky
● Log the version of the Dag Operator and Plugins when
they are run
● When catchup is enabled, new jobs will be added to
previous executions

56. scars of experience using xcom between jobs

57. scars of experience using xcom between jobs

58. scars of experience using xcom between jobs
● Alternatives
○ By default, the return value of the operator execute
method is stored in XCom
○ XCom values are stored in the Airflow metadata DB
○ Avoid using XCom
○ Store the state in data stores (databases, object
stores, etc)

59. scars of experience scheduling duration

60. scars of experience breaking changes
● Minor versions of Airflow can introduce breaking changes
○ Example: named parameter in S3Hook (1.8 -> 1.9)
■ aws_conn_id
■ s3_conn_id
Reference: Airflow development mailing list

61. where did all the magic of
machine learning workflows go?
Image credit: XKCD

62. have you ever
built machine learning pipelines?
no yes ?

63. Reference: TFX

64. Interest reading: Medium
airflow machine learning specifics
● Machine learning jobs are similar to usual jobs
● Factors which can affect the operator choice:
○ is the model built using the same Python version?
○ how much CPU and memory does your model need?
○ how can you make Airflow use your existing
infrastructure
○ how many concurrent workers do you need?
■ Limitation on scaling celery executors
■ Kubernetes executors in early stage

65. demo
Apache Airflow pipeline
(example of model building pipeline)

66. sample DAG model building pipeline

67. sample DAG model building pipeline

68. sample DAG model building pipeline content

69. sample DAG model building pipeline user data

70. sample DAG model building pipeline model

71. demo
Apache Airflow pipeline
(example of hyperparameter tuning)

72. sample DAG hyperparameter tuning

73. getting involved
Image credit: XKCD

74. airflow install
$ pip install apache-airflow

75. airflow source-code github
https://github.com/apache/airflow

76. airflow docs
https://airflow.apache.org/

77. airflow issue tracker jira
https://issues.apache.org/jira/browse?jql=project=AIRFLOW

78. airflow community slack
https://apache-airflow.slack.com

79. is Airflow the right tool for you?
Image credit: XKCD

80. http://datalab.rocks
find out more

81. дуже тобі дякую
Спасибо
Image credit: Wikipedia Commons
@tati_alchueyr