This document discusses Apache Airflow and its use at Dailymotion. It provides an agenda that covers data at Dailymotion, Apache Airflow, how Airflow is used at Dailymotion, deployment of Airflow at Dailymotion, working on a DAG (directed acyclic graph) pipeline, and an example pipeline for Dailymotion's new Advanced Analytics project. The example pipeline aggregates data from different sources with varying frequencies and timezones into BigQuery and Exasol for visualization in Tableau.

1. © 2018
Apache Airflow at Dailymotion
Wednesday20th June 2018

2. Cédric
Hourcade
Germain
Tanguy
• Data Engineer
• Github: https://github.com/germaintanguy
• Linkedin: https://fr.linkedin.com/in/germain-tanguy
• Data Engineering Manager
• Github: https://github.com/hc
• Linkedin: https://www.linkedin.com/in/hourcade/

3. Agenda
• Data at Dailymotion
• Apache Airflow
• Airflow at Dailymotion
• Deployment
• Working on a DAG
• Example of a pipeline

4. Data at
Dailymotion
© 2018

5. Data at Dailymotion
the team

6. Data at Dailymotion
who
what

7. Data at Dailymotion
who
tools

8. Apache Airflow
© 2018

9. Apache Airflow
Success
Retry
Failed
• Workflowmanagement system
• Workflow is defined as a Directed Acyclic
Grah (DAG)
• Each DAG is composed by task
• Workflowis defined in Python
• Schedulable
What airflow is?

10. Apache Airflow
What airflow is
solving?
• Scheduling dependanttasks
• Visualizing pipeline
• Monitoring / Alerting
• Backfilling
What airflow is
not?
• Data “streaming” :
Tasks do not move data from one to the other

11. Workflow
Command Line Interface:
• airflow list_dags
• airflow list_tasks
• airflow trigger_dag
• airflow test
• airflow clear
• airflow backfill ...

12. Architecture
Celery Executor

13. Operator / Sensor
Operatorbuilt-in:
• PythonOperator
• BashOperator
• BigQueryToBigQueryOperator
• GcsDataFlowJavaOperator
• ShortCircuitOperator
• HTTPOperator
• ... Sensorbuilt-in:
• HivePartitionSensor
• BigQueryTableSensor
• DatadogSensor
• S3PrefixSensor
• SqlSensor
• ...

14. Airflow at
Dailymotion
© 2018
Deployment

15. Deployment at Dailymotion
Our first deployment:
the LocalExecutor

16. Deployment at Dailymotion
Stage Prod

17. Deployment at Dailymotion

18. Deployment at Dailymotion
~$ make run
[] starting worker...
[] starting scheduler...
[] Starting webserver...
** Airflow is ready!
airflow@7ecb6a08559c:~$ |
Dev

19. Airflow at
Dailymotion
© 2018
Working on a DAG

20. Working on a DAG
Main property : idempotence

21. Working on a DAG
Example on a simple case :
We want to load some files from
storage to BigQuery table on a
daily basis.
Before loading data to
production partition, we want to
verify its quality (example :
number of row more or less the
same as last week).

22. Working on a DAG
Working on
a new DAG

23. Working on a DAG
Working on
a new DAG

24. 2
4
Working on a DAG

25. Airflow at
Dailymotion
© 2018
Example of a pipeline

26. Example of a pipeline
Presentation of one real use case : Dailymotion’s new Advanced Analytics
Goals :
• External : allowing partners to monitor theirs activities on Dailymotion, through a
serie of dashboards
• Internal : allowing internal business teams (ads op, finance, content..) to query
aggregated data via tableau
Challenges:
• Differents sources with different frequency of updated with
different timezone
• Backfill anything at anytime
• Monitor and alerting
• Solution approchable by differentdata friendly profession
Tools:
• Airflow
• BigQuery
• Exasol
• Tableau

27. Example of a pipeline
Presentation of one real use case : Dailymotion’s new Advanced Analytics

28. Workflow of a pipeline
DAG

29. Workflow of a pipeline
DAG

30. Workflow of a pipeline
DAG

31. Workflow of a pipeline
DAG

32. © 2017
Confidential
© 2018
Thank you
Cédric Hourcade et Germain Tanguy