InfoTrack: Creating a single source of truth with the Elastic Stack

Creating a single source of
truth with the Elastic Stack
A session with
Ashim Joshi - Head of Innovation
Hosted by

x
Intro
The problem
Validation + Roadmap
Datalake
Datamarts
API + Elastic
Final Product
Future Implementation
Conclusion
Idea to Reality

x
Intro
The problem
Datalake
Datamarts
API + Elastic
Final Product
Conclusion
Idea to Reality
We deal with huge volumes of activity daily
We initially monitored our IIS logs manually
using a ﬁle log tool.
This was horribly ineﬃcient, unscalable
and presented issues with transparency as
we couldn’t trace applications.
With the Elastic Cloud we had all of our pain
points answered. Elasticsearch could store
and search our data blazingly fast, Log-
stash transformed our data into a readable
format as we stream data into it and Kibana
would allow us to display our data in beau-
tiful customisable dashboards. The Elastic
Cloud has saved us months instead of
building an in-house solution, as well as
saved resourcing a dedicated developer to
maintain the product.

x
Intro
Idea to Reality
The problem
Datalake
Datamarts
API + Elastic
Final Product
Conclusion
Job titles aren’t binding. Each of us are equipped with a diverse skill
set, and we always leverage each other’s interests and strengths to
create better outcomes for our projects.
x3 Data Engineers
x1 Data Scientist
x6 Developers
x2 Designer
x1 Product Manager

X
Intro
The problem
Datalake
Datamarts
API + Elastic
Final Product
Conclusion
Idea to Reality

x
Intro
The problem
Datalake
Datamarts
API + Elastic
Final Product
Conclusion
DATAMARTS
Streaming
API Layer
Apps
DATA LAKE
Idea to Reality

x
Intro
The problem
Datalake
Datamarts
API + Elastic
Final Product
Conclusion
DATAMARTS
Streaming
Platform
Apps
DATA LAKE
Idea to Reality

x
Intro
The problem
Datalake
Datamarts
API + Elastic
Final Product
Conclusion
A data lake is a centralized repository that allows you to store all
your structured and unstructured raw data at any scale.
S3 bucket(s)
Idea to Reality

x
Intro
The problem
Datalake
Datamarts
API + Elastic
Final Product
Conclusion
S3 bucket(s)
structured data from
relational databases
(rows and columns)
Idea to Reality

x
Intro
The problem
Datalake
Datamarts
API + Elastic
Final Product
Conclusion
S3 bucket(s)
(rows and columns)
semi-structured data
(CSV, logs, XML,
JSON, Parquet)
Idea to Reality

x
Intro
The problem
Datalake
Datamarts
API + Elastic
Final Product
Conclusion
S3 bucket(s)
(rows and columns)
(CSV, logs, XML,
JSON, Parquet)
unstructured data
(emails, documents,
PDFs)
Idea to Reality

x
Intro
The problem
Datalake
Datamarts
API + Elastic
Final Product
Conclusion
S3 bucket(s)
(rows and columns)
(CSV, logs, XML,
JSON, Parquet)
unstructured data
(emails, documents,
PDFs)
binary data
(images, audio,
video)
Idea to Reality

x
Intro
The problem
Datalake
Datamarts
API + Elastic
Final Product
Conclusion
Why do we need Datalake?
Data has been stored everywhere in diﬀerent databases and services.
Idea to Reality

x
Intro
The problem
Datalake
Datamarts
API + Elastic
Final Product
Conclusion
It is getting harder and harder when we want to query our data.
Idea to Reality

x
Intro
The problem
Datalake
Datamarts
API + Elastic
Final Product
Conclusion
It is challenged to see overall picture of our business.
It is getting harder and harder when we want to query our data.
Idea to Reality

x
Intro
The problem
Datalake
Datamarts
API + Elastic
Final Product
Conclusion
Data lakes are next-generation hybrid data management solutions that can
meet big data challenges and drive new levels of real-time analytics from one
single source of truth.
Idea to Reality

x
Intro
The problem
Datalake
Data Source
Data Source
Amazon MSK
S3 Raw Data
Databricks
Streaming
Batch
Process
Lambda
Data ﬁles
Elastic Search
Data marts
SnowﬂakeS3 Processed Data
Datamarts
API + Elastic
Final Product
Conclusion
Our Datalake
Idea to Reality

x
Intro
The problem
Datalake
Datamarts
API + Elastic
Final Product
Conclusion
Geospatial
Entities
Time Series
Documents
Graphs
Customer Data
Idea to Reality

x
Intro
The problem
Datalake
Datamarts
API + Elastic
Final Product
Conclusion
Data mart
BI Finance MarketingOther Product teams
Data mart Data mart Data mart Data mart Data mart
Data Catalogs
Idea to Reality

x
Intro
The problem
Datalake
Datamarts
API + Elastic
Final Product
Conclusion
Data Source
Data Source
Amazon MSK
S3 Raw Data
Databricks
Streaming
Batch
Process
Lambda
Data Files
Elastic Search
Keywords
Translation API Layer
Details
S3 Processed Data
Idea to Reality

x
Intro
The problem
Datalake
Datamarts
API + Elastic
Final Product
Conclusion
Data Source
Data Source
Amazon MSK
S3 Raw Data
Databricks
Streaming
Batch
Process
Lambda
Data Files
Elastic Search
S3 Processed Data
Data marts
Translation API Layer
Idea to Reality

x
Intro
The problem
Datalake
Datamarts
API + Elastic
Final Product
Conclusion
What is Search-able Content
Idea to Reality

x
Intro
The problem
Datalake
Datamarts
API + Elastic
Final Product
Conclusion
Our clients ingesting data through our API’s. Pay per API request?
A scalable and performant platform to support our product, data and customer needs
Data anomaly detection in authority source data -> provide data that's more complete
and accurate than source data by combining and processing multiple sources.
Our clients developing their own products on their data
Single view of customer
Marketplace for data
Idea to Reality

Thank you
A session with
Ashim Joshi - Head of Innovation
Hosted by

InfoTrack: Creating a single source of truth with the Elastic Stack

Recomendados

Recomendados

Mais conteúdo relacionado

Mais procurados

Mais procurados (20)

Semelhante a InfoTrack: Creating a single source of truth with the Elastic Stack

Semelhante a InfoTrack: Creating a single source of truth with the Elastic Stack (20)

Mais de Elasticsearch

Mais de Elasticsearch (20)

Último

Último (20)

InfoTrack: Creating a single source of truth with the Elastic Stack