Data Architecture Best Practices for Advanced Analytics

1. Data Architecture Best Practices for Advanced Analytics Presented by: William McKnight “#1 Global Influencer in Big Data” Thinkers360 President, McKnight Consulting Group A 2 time Inc. 5000 Company @williammcknight www.mcknightcg.com (214) 514-1444 Second Thursday of Every Month, at 2:00 ET With William McKnight

2. Image Goes Here Know Better® May 12, 2022

3. © 2022 ChaosSearch, Inc. ChaosSearch Activates the Data Lake for Analytics at Scale The ChaosSearch Data Lake Platform eliminates complexity and helps organizations overcome the challenges posed by costly, siloed analytics solutions. Users perform both search and SQL queries concurrently and in situ from their cloud object storage, without data pipelining, transformation, or movement. The end result: Dramatic reductions in time, cost and complexity. FEATURED CUSTOMERS

4. © 2022 ChaosSearch, Inc. The ChaosSearch Approach: Connect Seamlessly, Eliminate Complexity Constantly Scale and Reduce Cost 3 * This is a roadmap item and subject to change. BENEFICIAL OUTCOMES ✓ One unified data lake for analytics at scale ✓ Combined search, SQL insights (PLG, security, etc.) ✓ No data pipelines or data movement ✓ No schema management, sharding or managing server clusters ✓ Simple, open API access to analytics tools of choice ✓ Scale and performance for analytic workloads with up to 80% cost savings ✓ Unprecedented simplicity

5. © 2022 ChaosSearch, Inc. Optimize Cloud Apps, Infrastructure and Security Optimize CloudOps/DevOps • Unlimited retention to optimize troubleshooting and performance of increasingly complex cloud architectures • Better log coverage to shorten time to resolution • Eliminate administrative toil, reduce operational costs 4 Optimize SecOps • Affordable long-term retention for in-depth forensics • Centralize logs in a security data lake for end-to-end visibility and monitoring • Simpler, more cost-effective compliance

6. “With the move to ChaosSearch, 98% of all operational burdens have been lifted from us, allowing us to focus on Blackboard-specific tasks.” Joel Snook, Director, DevOps Engineering

7. © 2022 ChaosSearch, Inc. 6 Single Access Pane on Global Deployment Case Study Challenge Need to reduce Elasticsearch costs & improve stability • Architectural challenges at scale caused system downtime requiring support • Challenges with infrastructure planning due to unpredictable log spikes -- especially during COVID-19 when demand grew 3,000% • SREs spent 10-15+ hours per week maintaining environment Solution • GDPR-compliant ChaosSearch log analytics solution provides single access pane to storage across multiple environments • True managed service with 99.999% availability • Use cases: visibility of cloud environments at scale, long-term app troubleshooting & alerting, root cause analysis Impact • Increased uptime with decreased management overhead • Retention more than doubled at 50% cost • No more data duplication • Querying activities scale on demand without manual reconfigurations or cost increases • SREs spend time on value-add work vs. ELK stack maintenance Need access to long-term data for resolving academic disputes, compliance purposes, etc.

8. © 2022 ChaosSearch, Inc. Before: Amazon ES Single Access Pane on Global Deployment © 2021 ChaosSearch, Inc. 7 Case Study • Multiple environments per region per team (3 teams) • Unstable environment with massive admin pain & multiple silos with failure points (master nodes) • $2M per year • 1-3 days retention AZ2 AZ1 … … ca-central-1 AZ2 AZ1 … … us-east-2 AZ2 AZ1 … … ap-northeast-2 AZ2 AZ1 … … eu-central-1 AZ2 AZ1 … … AZ2 AZ1 … Master Master Master Data Node Data Node Data Node Data Node Master Master Master Data Node Data Node Data Node Data Node Master Master Master Data Node Data Node Data Node Data Node Master Master Master Data Node Data Node Data Node Data Node Master Master Master Data Node Data Node Data Node Data Node Master Master Master Data Node Data Node Data Node Data Node … ap-southeast-2 ap-southeast-1 With ChaosSearch Single access pane, multiple environments for storage & access isolation (e.g., GDPR) $950k per year (50%+ discount) True managed service with 5 9s availability 7 days retention (could be unlimited)

9. © 2022 ChaosSearch, Inc. A Question for Our Audience Today… 8 What are your greatest challenges when it comes to deriving insight from all your data? • Expertise • Resources • Technology • Time • Cost • Other

10. Thank you! Learn more ● ChaosSearch.io ● Free Trial - chaossearch.io/trial ● Demo Webinar - chaossearch.io/demo-webinar ● 2022 Cloud Data & Analytics Survey Report

11. © 2022 ChaosSearch, Inc. 10 Connect to any and all data in your cloud object storage Index into highly compressed, unified representation of data that never leaves your storage Prepare your data views for discovery and analytics with no data movement Use your existing tools for • Log analytics • Operational BI queries • Dashboards • Monitoring and alerting ChaosSearch is Built for Operational Analytics at Scale Index 2 Refine 3 Analyze 4 Store 1

12. © 2022 ChaosSearch, Inc. Log Analytics Transformed Before: Elasticsearch (ELK stack) DevO ps SecO ps LOB ??? • Limited retention • Expensive to scale • Management and configuration challenges • Downtime created by instability at scale • Multiple data silos created due to the limits above Cloud Object Storage i.e., Google GCS, AWS S3 Dev Ops Sec Ops LOB ??? PUBLISHED ELASTIC API One unified data lake Unlimited scale and retention. Save up to 80% on Managed Service with 99.99% uptime. With ChaosSearch 11

13. Appendix

14. © 2022 ChaosSearch, Inc. 13 Partially In-Situ architectures do work and add incremental value. But fall short of a true breakthrough. In-Situ INFORMATION In-Situ PROCESSING • Underlying representation of data is native to use case • Multi-model access • Distributed processing • Leveraging the power of cloud computing elasticity In-Situ ALGORITHM • Low memory execution • Stream based operations A fully In-Situ Architecture is needed to unlock the full spectrum of In-Situ advantages

15. A Unified Data Lake Architecture for Log and SQL Analytics 14 ChaosSearch uniquely solves for known and unknown data and queries * Source: Gartner. DW = data warehouse The Zone of Confusion Within the Data and Analytics Infrastructure Model Expanding Understanding and Investigating Founational Core Innovation and Exploration Establish Value Traditional DW Data Lake Zone of Confusion Questions Known Unknown Data Unknown Known Bring together search and relational analytics • Eliminate pipelines, ETL, data movements • Faster insights A unified data lake architecture that supports: • Innovation & exploration • Investigative queries • Operational analytics Disrupts the economics of Big Data

16. A Unified Data Lake Architecture for Log and SQL Analytics 15 ChaosSearch uniquely solves for broadest scope of analytics needs Bring together search and relational analytics • Eliminate pipelines, ETL, data movements • Faster insights A unified data lake architecture that supports: • Innovation & exploration • Investigative queries • Operational analytics Disrupts the economics of Big Data * Source: Gartner. DW = data warehouse The Zone of Confusion Within the Data and Analytics Infrastructure Model Expanding Understanding and Investigating Founational Core Innovation and Exploration Establish Value Traditional DW Data Lake Questions Known Unknown Data Unknown Known

17. William McKnight President, McKnight Consulting Group • Consulted to Pfizer, Scotiabank, Fidelity, TD Ameritrade, Teva Pharmaceuticals, Verizon, and many other Global 1000 companies • Frequent keynote speaker and trainer internationally • Hundreds of articles, blogs and white papers in publication • Focused on delivering business value and solving business problems utilizing proven, streamlined approaches to information management • Former Database Engineer, Fortune 50 Information Technology executive and Ernst&Young Entrepreneur of Year Finalist • Owner/consultant: Data strategy and implementation consulting firm William McKnight The Savvy Manager’s Guide The Savvy Manager’s Guide Information Management Information Management Strategies for Gaining a Competitive Advantage with Data 2

18. 3 All Data Under Management Best Practice: Get all data under management

19. Data is Under Management when it is… • In a leveragable platform • In an appropriate platform for its profile and usage • With high non-functionals (Availability, performance, scalability, stability, durability, secure) • Data is captured at the most granular level • Data is at a data quality standard (as defined by Data Governance) • Enables self-service 4 Best Practice: Enpower everyone with true self-service

20. “80% of analysts’ time is spent simply discovering and preparing data.” What’s Your Data Strategy, Thomas Davenport, HBR 2017 Best Practice: Start getting concerned with the tools and processes of the analyst

21. The Relational Database Data Page © McKnight Consulting Group, 2010 Page Header Page Footer Row IDs Records 1120MCG William McKnight President 214-514-1444 wmcknight@mcknightcg.com 1121Stolt Offshore MS Ltd Joe Tyron Director 226-5555- 1269 jtyron@stoltoffshore.com 1122Medtronic, Inc. Mark Smith Principle Database Administrator 763-555-2557 mark.smith@medtronic.com

22. Columnar Orientation 7 Best Practice: Make all analytic structure columnar

23. Data Lakes • P 8 Parquet format Best Practice: Put big data in data lakes Best Practice: Index the data lake

24. Data Lakes • Common & centralized storage for the enterprise • No defined data model into which the data is formed • No relationships between the datasets • Historical data retention • All data formats • For big data • Analytical processing • Data scientists and analysts • Less governance/quality than data warehouse – Focus: Ingestion 9

25. Graph Databases Bridge vertex Bridge vertex 10 • Subject: John R Peterson Predicate: Knows Object: Frank T Smith • Subject: Triple #1 Predicate: Confidence Percent Object: 70 • Subject: Triple #1 Predicate: Provenance Object: Mary L Jones Best Practice: Use graph databases for sizable connected data

26. Data Virtualization “The right answer is not always to centralize the data. Data Virtualization will be of utmost importance as the ‘perpetual short-term’ solution to the need.” 11 Data Warehouses Marts & Cubes Operational Data Stores Transactional Sources File Systems Big Data Enterprise Data Virtualization Best Practice: Enable data virtualization for edge and temporary needs

27. Enterprise Analytic Stack • Dedicated Compute • Storage • Data Integration • Streaming • Analytics • Data Exploration • Data Lake • Business Intelligence • Machine Learning • Identity Management • Data Catalog • Data Virtualization Best Practice: Leverage best of breed for your analytics stack

28. • Autonomous Administration • Lack of Platform Features Leads to Increased Configuration and Management – stored procedures, referential integrity and uniqueness capabilities – mission critical options for backup and disaster recovery, which typically includes a standby database – full ANSI-SQL compliance • Performance Total Cost of Ownership is More Than Just Cloud Costs Best Practice: Get a strong handle on your cloud costs

29. Capabilities for Data Integration for Enterprise Data • Comprehensive Native Connectivity • Multi-Latency Data Ingestion • Data Integration (in ETL, ELT, Streaming) • Data Quality and Data Governance • Data Cataloging and Metadata Management • Enterprise Trust, Enterprise Scale (or Class) • AI Intelligence and Automation • Ecosystem and Multi-cloud

30. Data Integration Options Project Technical Environments Recommended For Consideration Project Scope Heterogenous: Cloudera Any Any IBM Any Any Informatica Any Any Talend Any Any Specialist: AWS (Glue) Environments on AWS with core of Redshift, EMR Any Azure (Azure Data Factory) Environments on Azure with core of Synapse, HDInsight Any FiveTran Any Contained scope Google Environments on GCP with core of BQ, DataProc Any Matillion Any Contained scope Oracle Environments with Oracle database Any SAP SAP-only environments SAP projects Best Practice: Use fit-for- purpose data integration

31. Competitive Analytic Architectures

32. Architecture Component Needs • Security and Privacy • Governance and Compliance • Availability • Backup and Recovery • Performance • Scalability • Licensing 17

33. Analytics Reference Architecture Logs (Apps, Web, Devices) User tracking Operational Metrics Offload data Raw Data Topics JSON, AVRO Processed Data Topics Sensors and / or Transactiona l/ Context Data OLTP/ODS ETL Or EL with T in Spark Batch Low Latency Applications Files In- database analytics Reach through or ETL/ELT or Stream Processing or Stream Processing Q Q Data Warehouse Data Lake

34. Data Lakehouse 19 and / or In- database analytics Reach through Q Q Data Warehouse Data Lake

35. Data Mesh Logs (Apps, Web, Devices) User tracking Operational Metrics Offload data Sensors and / or Transactiona l/ Context Data OLTP/ODS Batch Low Latency Applications Files In- database analytics Reach through or ETL/ELT or Stream Processing or Stream Processing Q Q Data Warehouse Data Lake ETL Or EL with T in Spark Raw Data Topics JSON, AVRO Processed Data Topics

36. Data Fabric Logs (Apps, Web, Devices) User tracking Operational Metrics Raw Data Topics JSON, AVRO Processed Data Topics Sensors and / or Transactiona l/ Context Data OLTP/ODS ETL Or EL with T in Spark Batch Low Latency Applications Files In- database analytics or Stream Processing Data Warehouse Data Lake

37. Best Practice: Pursue mesh and fabric architectures to the degree possible

38. Data Cloud (Snowflake) Logs (Apps, Web, Devices) User tracking Operational Metrics Raw Data Topics JSON, AVRO Processed Data Topics Sensors and / or Transactiona l/ Context Data OLTP/ODS ETL Or EL with T in Spark Batch Low Latency Applications Files In- database analytics or Stream Processing Data Warehouse Data Lake

39. Summary • Get all enterprise data under management • RDBMS (/columnar), Cloud Storage/Parquet, Graph cover most analytic platform needs • Cost of ownership is more than the cloud costs • Data Integration is vital to Data Architecture for Modern Analytics • The Data Mesh and Data Fabric are decentralizing the architecture 24

40. Upcoming Topics • Is Our Information Management Mature? • The Future based on AI & Analytics • Organizational Change Management for Data & Analytics Driven Projects • Graph Database Use Cases • Assessing New Databases: Translytical Use Cases 25 Second Thursday of Every Month, at 2:00 ET

41. Data Architecture Best Practices for Advanced Analytics Presented by: William McKnight “#1 Global Influencer in Big Data” Thinkers360 President, McKnight Consulting Group A 2 time Inc. 5000 Company @williammcknight www.mcknightcg.com (214) 514-1444 Second Thursday of Every Month, at 2:00 ET #AdvAnalytics

Data Architecture Best Practices for Advanced Analytics

Data Architecture Best Practices for Advanced Analytics

Recomendados

Mais conteúdo relacionado

Mais procurados

Mais procurados(20)

Similar a Data Architecture Best Practices for Advanced Analytics

Similar a Data Architecture Best Practices for Advanced Analytics(20)

Mais de DATAVERSITY

Mais de DATAVERSITY(20)

Último

Último(20)

Data Architecture Best Practices for Advanced Analytics