This presentation explains what data engineering is and describes the data lifecycles phases briefly. I used this presentation during my work as an on-demand instructor at Nooreed.com

1. Introduction to Data Engineering
Hadi Fadlallah

2. About Me
…
…
…
…
Just a Data Engineer!
4/6/2022 2

3. Plan
• What is Data Engineering?
• Data Engineer vs. Data Scientist vs. Data Analyst
• Understanding Data Management (Data Layers, DQS, MDS,
Provenance)
• Distributed Computing
• Designing Data Pipelines (Choosing Paradigm / Technologies)
• Data Engineer Jobs / Required Skills
• Helpful Tips
• Online Courses
4/6/2022 3

4. What is Data
Engineering?
4/6/2022 4

5. What is Data Engineering?
4/6/2022 5
• Data Engineering is the act of:
• Collecting data
• Transforming (…) data
• Validating data
“Making data consumable”

6. What is a Data Engineer?
AI/ML Engineer BI Developer Data Analyst
Database
Administrator
Report Developer Data Developer
Data Architect Data Integration
Specialist
ETL Developer
Data Scientist
4/6/2022 6

7. Data Engineer vs. Data Scientist
4/6/2022 7
Source: https://elu.nl/careers-in-data-science-data-analyst-vs-data-engineer-vs-data-scientist/

8. 4/6/2022 8

9. 4/6/2022 9
Data
Sources
Processed
Data
Storage
Data
Ingestion
Raw Data Storage
Data Processing Data
Visualization
Understanding Data Layers

10. 4/6/2022 10
Source: https://www.sharpersoftware.com/data/dqs

11. 4/6/2022 11
Source: https://www.dataentryoutsourced.com/blog/ever-changing-master-data-management/

12. Data Provenance
4/6/2022 12
Source: https://www.canto.com/blog/data-lineage/

13. Data Wrangling vs. Data Pre-processing
4/6/2022 13
Source: https://medium.com/swlh/data-pre-processing-data-wrangling-4a6a8624e747
Data Pre-processing Data Wrangling

14. What are Distributed Systems?
4/6/2022 14

15. 15/20
• 1 Cluster = many
machines (nodes)
• Parallel computing
• Scalability
• Fault-tolerance (High
availability)

16. 16/20
Data ingestion Data storage Data Processing

17. 4/6/2022 17
Designing Data
Pipelines

18. Understanding Data &
Requirements
4/6/2022 18
Source:
https://firebrand.training/uk/lea
rn/pmp/course-material/project-
scope-management/collect-
requirements

19. Choosing a Paradigm
4/6/2022 19

20. 4/6/2022 20
Source: https://www.xplenty.com/blog/etl-vs-elt/

21. 4/6/2022 21
Source: https://www.quora.com/Why-and-when-should-I-use-NoSQL-instead-of-SQL

22. 4/6/2022 22
Source: https://www.talend.com/resources/customer-360-data-hub/

23. Choosing the RIGHT
Technologies
4/6/2022 23

24. Relational databases
4/6/2022 24
Source: https://www.stambia.com/en/solutions/by-technology/integration-to-database/relational-database-rdbms

25. Data Integration
4/6/2022 25
Source: https://deevita.com/top-data-integration-tools/

26. NoSQL databases
4/6/2022 26
Source: https://dock.ie/data.html

27. 4/6/2022 27
Big Data ecosystem
Source: https://sranka.wordpress.com/2014/01/29/big-data-technologies/

28. 4/6/2022 28
Data Engineer Jobs

29. 4/6/2022 29
Stack Overflow: https://stackoverflow.com/jobs?q=data+engineer
LinkedIn: https://www.linkedin.com/jobs/search/?geoId=92000001&keywords=data%20engineer&location=Remote

30. Required Skills
2. SQL, Python
3. Pandas library
4. AWS, Azure
5. Relational databases (SQL Server and MySQL)
6. Hadoop, Spark, Hive
7. NOSQL databases (MongoDB, Neo4j, and Cassandra)
…
4/6/2022 30

31. 4/6/2022 31
1st Required Skill

32. 4/6/2022 32
Source: https://towardsdatascience.com/most-in-demand-tech-skills-for-data-engineers-58f4c1ca25ab

33. 4/6/2022 33

34. 4/6/2022 34
First, make it run
correctly (in an
acceptable way). Then
start improving
performance

35. 4/6/2022 35
Again, make sure
that you understand
well the data and
the requirements!

36. 4/6/2022 36

37. DON’T use Big Data
technologies just
because you have a
huge volume of
data.
Follow best
practices
4/6/2022 37

38. 4/6/2022 38
Online
Courses

39. 4/6/2022 39
• Coursera:
• Google Cloud - Data Engineering, Big Data, and Machine Learning on GCP
Specialization
• San Diego - Big Data Specialization
• Udacity:
• Data Engineering nanodegree
• DataCamp:
• Data Engineer with Python Track
• IBM – CognitiveClass.ai
• Free data science and data engineering courses
• Udemy:
• Data Science A-Z™: Real-Life Data Science Exercises Included

40. Thank you
4/6/2022 40
https://hadifadl.github.io