Data Observability.pptx

Pallavi Angraje, Sona Samad
Data Observability : The
Force Awakens
Oct 14, 2022

©2022 Intuit Inc. All rights reserved. 2
About Us
Intuit Data Ecosystem: Unique consumer and small business assets at scale
Data & Analytics at Intuit
Product Reports and Analytics Dashboards
Data Quality Challenges
Understanding data issues
Data Observability
Cure, Detect, Prevent, Eradicate : Data Observability Model in Intuit
Achieving Data Quality At Scale
Preventing Data Incidents Using DQ Checks, ADS & Infrastructure Monitoring
Agenda

Intuit Data Ecosystem: Unique consumer and
small business assets at scale

Intuit Product Report

The monthly aggregated metrics
for subscribers is not matching
with the weekly data.
Are there any issues?
-Lisa (Frustrated Data Worker)

Data Sources
Do not understand how
downstream uses the data
How the challenges originate?
Data Lake
Trouble identifying the ways the
data pipelines might break
Analytics
Hard to understand what’s wrong
in the data and how to get some
help

What is not well ?
Incorrect Reporting
More Incidents
High MTTD/
MTTR
Missed SLA
Data
Reload
Parity
Issues

Event
Parity
Issues
Source
Failures /
Delays
Data /
Processing
Defects
Prediction
Model
Defects
Incorrect
Reporting
SLA Misses
Data Analytics Platform

Achieving Data Quality
at Scale.

Let’s Recall the Problem
Incorrect Reporting
More Incidents
High MTTD/
MTTR
Missed SLA
Data
Reload
Parity
Issues

Cure Detect Prevent Eradicate
Monitoring Dashboard Alerting Protocol

Preventing Data Incidents: Data Quality Checks
Circuit
Breaker
Checks
Low
Priority
Alerts
Data
Validation
Checks
Key Design Decisions
● Multiple Source Support
● Performance Consistency & Scale
● Config Driven Data Profile Rules
● Capability to add Business Rules
● Run as part of Data Processing Pipelines
● Data Discrepancy and Anomaly Detection
● Fail Fast with Circuit Breaker
● Multi-channel Alerts, Single Window Reporting
Data
Pipeline 1
High
Priority
Alerts
Data
Pipeline 2
Data Quality Checks

Data
Quality
Library
Data
Source
Reports /
Alerts
Data
Quality
Rules
Data
Quality
Library
Data
Quality
Rules
Reports /
Alerts

DB file
.parquet
.csv
.xml
.json
Input
Sources
Object Store
Spark
SQL
Spark
Config
Input
Configs
Alerts
Triage
Dashboard
DB
logs Output
Object Store
Spark Process
Dataframe 3
Dataframe
Comparator
Spark Loader
Class
Spark Loader
Class
Dataframe 1 Dataframe 2
Spark Step 1 Spark Step 2
Spark Loader
Class
Dataframe n
Dataframe
Validator
Spark Step n

DB
logs Output
Object Store
DB file
.parquet
.csv
.xml
.json
Input
Sources
Object Store
Spark
SQL
Spark
Config
Input
Configs
Alerts
Triage
Dashboard
Spark Process
Dataframe 3
Dataframe
Comparator
Spark Loader
Class
Spark Loader
Class
Dataframe 1 Dataframe 2
Spark Loader
Class
Dataframe n
Dataframe
Validator
Spark Step n
Spark Loader
Class
Load
Transform
Save
dataframe
dataframe
PreStep = {
class-name = com.intuit.MySparkLoadClass,
inputs = {
my-company-gns-df = {
order = 2, sql = {
sql-type = local,
out-path = "s3://temp/path", table = companies_filtered,
sql = """select a.* from demo.company_info a
join demo.company_status b on a.c_id = b.c_id
where instr(a.company_name, 'delete') = 0"""
}, metadata = {is-input = true, is-save = true}
}
}
Custom Scala Class
Custom Spark SQL

DB file
.parquet
.csv
.xml
.json
Input
Sources
Object Store
Spark
SQL
Spark
Config
Input
Configs
Alerts
Triage
Dashboard
Spark Process
Spark Loader
Class
Spark Loader
Class
Spark Loader
Class
Dataframe n
Dataframe
Validator
Spark Step n
Spark Loader
Class
compare_1 = {
class-name = com.intuit.Dataframe
Comparator
,
properties = {
"comparator-config" = """{
"comparisonSets" :
[{"Product":"df_dataset_1"},
{"Billing
System":"df_dataset_2"}],
"validationName": "Test QBO
signup count by product ",
"comparisonType":
"percent_variance",
"threshold":"1.00",
"comp_out_df":
"df_gns_by_product"
}"""
}
Custom Scala Class
Comparison Sets
Validation Name
Threshold value
Output Dataframe
Comparison Type
config input
comp_out_df
Dataframe
Comparator
df_dataset_1 df_dataset_2

DB file
.parquet
.csv
.xml
.json
Input
Sources
Object Store
Spark
SQL
Spark
Config
Input
Configs
Alerts
Triage
Dashboard
Spark Process
Spark Loader
Class
Spark Loader
Class
Spark Loader
Class
Dataframe n
Dataframe
Validator
Spark Step n
Spark Loader
Class
comp_out_df
Dataframe
Comparator
df_dataset_1 df_dataset_2
DB
logs Output
Object Store
DatamartValidation = {
class-name = com.intuit.DataframeValidator,
properties = {
"validation-config" = """{
"pipelineName":"Test",
"validationResultSets" : ["my-company-gns-df",
"df_gns_by_product",..],
"resultGenericColumns":["validation_name:string",
"dimension_1:string","dimension_2:string",
"dataset_1:string","metric_1_value:decimal(20,4)",
"dataset_2:string","metric_2_value:decimal(20,4)",
"validation_result:string",
"is_valid:boolean"],
"outputDirectory":"s3://temp/outpath",
"emailId":"alertsmail@intuit.com",
"validatorOptions":{"failJob":"1", "emailSubject":"Test
Triangulation","forwardToSplunk":"1"}
}"""
}
}
Validation
Result Sets
Custom Scala Class
Result Set
Columns
Alert Email

Preventing Data Incidents: Anomaly Detection Checks
Features
● Ensemble of machine learning
algorithms
● Training done from historic
patterns
● Supports time series data
● Scheduled and API based
triggers for training and
inference
● Post inference anomalies
published for consumption
Time Series
Dataset
Majority Voting
Anomaly /
Not Anomaly

Data Anomaly Detection

Data Triage - Validating Data at Each Layer

Spark - Performance Optimization
Spark Lens
Ganglia Chart
Spark History Server

Preventing Data Incidents at 3 layers

Parity checks
to identify
event loss
Source Job
Failure / Delay
Alerts
Resource
Health
Monitoring Anomaly
Detection for
Data Outliers
Data Analytics Platform
Improved Forecast
Models
Data Quality
Checks &
Circuit
Breakers

Now I trust
the data !
Let’s ask Lisa

Data Observability.pptx

Recomendados

Recomendados

Mais conteúdo relacionado

Mais procurados

Mais procurados (20)

Semelhante a Data Observability.pptx

Semelhante a Data Observability.pptx (20)

Último

Último (20)

Data Observability.pptx