Synopsis: Modern enterprises anticipate business requirements and work proactively to optimise the outcomes. If they don’t renovate or reinvent their data architectures, they lose customers, and market share. So my talk will be in detailing the importance of data architecture, architectural challenges if is not addressed and a case study - the learnings and success story by fixing the issues at the root - at the data storage & access.
Target Audience: Principal Software engineers & Architects
Key Takeaways: Importance of Modern Data Architecture, PostgreSQL & JSONB
I have given a talk @ https://hasgeek.com/rootconf/elasticsearch-users-meetup-hyderabad/
3. AgendaData - The Core Why Data Architecture?
Business Use Case
Lessons Learnt
The Journey & Success Story
4. Data - The Domain
Films Character ActorDirector Drama Comedy Thriller
Patient Doctor Operation Disease Pills Visit
Video
Health Care
Symptom
Rom
Nurs
6. The Data
HBR's Alarming Survey Results
survey participants: 64 c-level technology and business executives representing
large corporations such as American Express, Ford Motor, General Electric,
General Motors, and Johnson & Johnson
2019 Big Data and AI Executive Survey
7. The Data Architecture
Data architecture describes how data is collected, stored,
transformed, distributed, and consumed
- Wikipedia
8. Data Architecture - A (Legacy) Sample
More
Data Sources
3rd Party APIs
Databases,
DWH & Lakes
Different Files
NoSQL
Application (or) Business Processing Data Consumers
More
Data Visualisers
3rd Party Apps
In House Apps
Files
Chaos
Calls for Renovation/Reinvention
9. Message Bus
Data Consumers
More
Data Visualisers
ML Tools
Applications
Microservices
More
Data Sources
3rd Party APIs
Databases,
DWH & Lakes
Different Files
NoSQL
Data Architecture - A (Modern) Sample
Application (or) Business Processing
10. Core Enablers of Modern Business
Solid Data Architecture - The Why?
IoT AIBig Data Cloud Native ML
Robust Data Architecture
11. Client Portfolio
Business Use Case
Through-Channel Marketing Automation (TCMA)
SaaS platform
➔ Accelerates Brand sales in Local Market
➔ Shares comprehensive Analytics on Individual
Tactics & Campaigns
➔ Helps Brands to adapt to market change
based on data driven insights
12. The (Domain) Data
Business Use Case
CampaignsRetailers Target Audience
Order Snapshots
Brands
Orders
and so on...
13. The Data Analysis
Business Use Case
The target Audience, the orders, the order
snapshots all capture different data points
per group of records
Chess List
➔ Classic FIDE
➔ Blitz FIDE
➔ Rapid FIDE
Football List
➔ Height
➔ Weight
➔ Nationality
Wrestling List
➔ Ring Name
➔ Height
➔ Weight
➔ Debut
15. R1_Orders R1_Snapshots R1_XYZ Indices
R1_Audience
_1
R1_Audience
_2
R1_Audience
_3
Indices
R1_Audience
_4
R1_Audience
_5
R1_Audience
_N
Indices
R2_Orders R2_Snapshots R2_XYZ Indices
R2_Audience
_1
R2_Audience
_2
R2_Audience
_3
Indices
R2_Audience
_4
R2_Audience
_5
R2_Audience
_N
Indices
R3_Orders R3_Snapshots R3_XYZ Indices
R3_Audience
_1
R3_Audience
_2
R3_Audience
_3
Indices
R3_Audience
_4
R3_Audience
_5
R3_Audience
_N
Indices
RN_Orders RN_Snapshots RN_XYZ Indices
RN_Audience
_1
RN_Audience
_2
RN_Audience
_3
Indices
RN_Audience
_4
RN_Audience
_5
RN_Audience
_N
Indices
450K Tables for 4K Customers -
Scalable?
Chaos.. Calls for Renovation!!
The Challenges
Business Use Case
16. The Reinvention
Business Use Case
Microservices
Netflix has fixed the Scalability Issues by adopting
microservices architecture. So, why not us?
NoSQL
No need to create multiple tables to support dynamic columns
instead, let us leverage NoSQL schema-less approach
By Client’s Labs Team
Is the foundation (data architecture) robust?
17. A Business Unit of
The solution
Business Use Case
proposed by
Cloud Native architecture with java microservices
Phase#2: Reinvention
Hybrid Data Architecture - Schema & Schemaless approach
Phase#1: Renovation
18. Renovation Highlights
Business Use Case
BRAND LIST FIRST_NAME LAST_NAME CUSTOM_DATA
B1 CHESS Vishwanathan Anand {“blitz”: 2786, “rapid”:2737}
B1 CHESS Garry Kasparov {“classic”: 2812, “blitz”: 2801, “rapid”:2783}
B2 FOOTBALL Christiano Ronaldo {“nationality”: “Portugal”, “height”: “1.87M”}
B2 FOOTBALL Lionel Messi {“nationality”: “Argentina”, “height”: “1.7M”}
B3 WRESTLING Geetha Phogat {“coach”: “Mahavir Singh”, “height”:”1.62M”}
#1 No Multiple Tables & Indices - Dynamic Columns using JSONB & GIN
20. A Business Unit of
Renovation Benefits
Business Use Case
proposed byNo Tables & Indices on the fly
➔ Instead Data was Organised for effective Storage
& quick Access
➔ Able to accomodate more Brands with the same
infrastructure
#2: Maximum Scalability
➔ The data architecture issues were fixed
➔ No New Tech Stack introduction, so not much
learning curve & additional deployment effort
#1: Minimal Disruption
24. Thank You!
Kayalvizhi Noorul Ameen, Principal Architect @ Pramati Technologies
Image Credit: unsplash.com & flaticon.com
The Case Study - published in Imaginea