The quality of business decisions, machine learning insights, and executive reports depend on the quality and integrity of the underlying data. There are many ways that data can get corrupted in an analytical data platform from de-synchronization with the system-of-record to defects in data pipelines. We will show how to detect and prevent data corruption with automation, open-source tools, and machine learning.

1. 2
Data Quality
Automatic enforcement in real-time with machine learning.
Max Martynov, CTO

2. Introducing Grid Dynamics technology services
Digital transformation Big data, real time analytics, ML & AI
Microservices replatforming DevOps & cloud enablement
Open Source Cloud-ready Scalable Automated

3. 12 years of
experience in digital
transformation.

5. 9.8
9.3 9.6 9.4
10.1
17.5
16.9
8.9 9.6 9.2
10.3
9.8
5.1
4.1 3.9 4 3.9
4.5
7.5 7.1
3.8 4.1 4 4.5
4.2
2.3
0
5
10
15
20
7/1/19 7/2/19 7/3/19 7/4/19 7/5/19 7/6/19 7/7/19 7/8/19 7/9/19 7/10/19 7/11/19 7/12/19 7/13/19 7/14/19
Retailer X, Daily Sales – Executive Summary
Revenue, $M Gross Profit, $M
weekend weekend

6. DBDBDB
EDW

7. Data Lake
EDW
DBDBDB FileFileFile

8. Cloud
Data Lake
EDW
DBDBDB FileFileFile MQ CloudCloud
AppAppAPI

9. Cloud
EDW
DBDBDB FileFileFile MQ CloudCloud
AppAppAPI
AppAppApp
Data Lake

10. 1 0 1 1
0 1 1 0
1 0 1 0
1 0 0 1
1. Trust is hard to build and easy to lose.
2. Distrust in data slows down decisions.
3. Slow decisions prevent agility.

11. Data corruption reasons
1. Code 2. Data Sources 3. Infrastructure

12. Test environment
Input
Actual
Expected
ETL
code
compareTest suite
run test
Traditional approach to testing

13. Production data quality goals
Detect
data corruption
Prevent
it from spreading
Alert
support team

14. Production data lake
Data quality enforcement in production
DBDBDB
FileFileFile
MQ
AppAppAPI
data processing job

15. Production data lake
DBDBDB
FileFileFile
MQ
AppAppAPI
data processing job data quality job
Data quality enforcement in production

16. Production data lake
DBDBDB
FileFileFile
MQ
AppAppAPI
data processing job data quality job
alert &
stop pipeline
alert &
continue
x
Data quality enforcement in production

17. Data Lake
Data
source
data
1. Compare with
SoR
2. Validate
business rules
3. Data profiling and
anomaly detection
Main data processing pipeline
confidence
data
confidence

18. 1. Control divergence from SoR
Data Lake
Data
source
Imported
dataset
Compare data in
SoR and data lake
1. Validate correctness of
import.
2. Prevent stale data.
3. Prevent corruption
accumulation in stream
processing use cases.
4. Check data before it gets in
the lake.

19. 2. Validate business rules
Data Lake
Dataset
Check for nulls and
data ranges
1. Enforce schema.
2. Check for nulls.
3. Validate data ranges.
4. Specify and enforce
data invariants.

20. 3. Anomaly detection
Data Lake
Dataset
1. Fully automatic data
quality enforcement.
2. Collect data profile,
metrics and statistics.
3. Train ML models.
4. Find anomalies in data.
Data profiling and
anomaly detection

21. Catalog
Inventory
Orders
Data Lake Data Quality
Reporting &
Alerting
Data Profile
Demo setup

22. 23
Live demo

23. Anomaly detection example

24. Anomaly detection example: zooming in

25. Capabilities for enterprise data quality and governance
 Enables widespread adoption.
 Enforces enterprise-level controls
and data usage policies.
 Increases consistency and
confidence in decision making.
 Decreases the risk of regulatory
fines.
 Improves data security.
 Facilitates accountability for
information quality.
 Minimizes or eliminates efforts
duplication.
Data Governance Platform
Metadata
Management
Full-text
Search
Data Quality
Status
Schema /
Summary
Data
Profiling
Mapping
to
Glossary
Change
Log
Dependency
Detection
Consumers Flow Visualization
Glossary Portal
Knowledge
Base
Fields
Fingerprinting
Data Catalog
Dataset Profile
Lineage Dashboard
Data
Glossary
Data Quality
Access and
Security
Business
Rules
Anomaly
Detection
Alerting
Access Rules
Compliance
Policies
Policy Engine

26. www.griddynamics.com
Thank you!

27. 28
Demo screenshots

28. Dataproc cluster

29. Aifrlow pipeline

30. Griffin measures

31. Anomaly detection: normal data

32. Anomaly detection: anomaly

33. Anomaly detection: return to normal

34. Anomaly detection: historical view

35. Anomaly detection (counts): anomaly & return

36. Uniqueness: normal data

37. Uniqueness: anomaly

38. Uniqueness: return to normal

39. Uniqueness: historical view

40. Nulls: normal data

41. Nulls: anomaly

42. Nulls: return to normal

43. Nulls: historical view

44. Ranges: historical view

45. Completeness: anomalies