1. CI/CD for a Data Platform
How to enable consistent data pipelines
2. 2
Your Host
| Koen Rottiers
| Senior Consultant @ Codit
| 9 years in IT, track record in networking
and infrastructure
| Combining people, business and
technology
CI/CD for a Data Platform: How to enable consistent data pipelines
@KoenRottiers
3. Agenda
| A Data Platform?
| What is Azure Data Factory?
| The Data Lake architecture
| Why do CI/CD for a Data Platform?
| Azure Data Factory Git integration
3
5. Data Platform overview
5
| Ingestion different sources
| Centralized data store
| Data flows through
| Output curated data
| Multiple inputs and
outputs
13. Data Platform Roles and Responsibilities
13
- Data platform owner: This person is the owner and responsible of the overall data platform.
- Data platform operator: This role is responsible for the day to day operational tasks of the platform
- Data pipeline owner: Different pipelines will be running on the platform. Each pipeline will have its own
purpose and so it’s specific owner. This is someone from the BI Team or business.
- Data pipeline developer: This person will be developing new pipelines or making adjustment to
existing ones.
- Data source owner: Different data sources will be integrated with the data platform. Every data source
will need to have an owner to determine access rights, access manner,... This person will be responsible
for the data residing in the source system. Most of the time this will be the application owner of the
application that uses the data source.
14. Key Advantages
14
| Consistent deployment of data pipelines
| Full testing of data flows in the Data Lake
| Better collaboration
| Feature development tracking
| Pipeline quality reviews
| More fine-grained data security
| Tracking data movements
23. So why?
23
| Let data engineer/data scientists focus on delivering value and insights to the
business
| Enable an agile process in data engineering
| Consistency across environments
| Track feature development / Bug fixing
| Be able to audit your data streams
24. Do you want a demo?
Feel free to reach out to us.
24