DIMT '23 Session_Demo_ Latest Innovations Breakout.pdf

2. Building streaming data pipelines with Conﬂuent’s latest innovations Engin Cukuroglu, PhD Manager - Solutions Engineering

3. Intro Understand Confluent’s product strategy 01 Recent Product Updates 12 month summary of Confluent launches 02 Demo Confluent Cloud data streaming in action 03 Session Agenda Confluent Roadmap See the future of cloud-native data streaming 04 Getting Started Start your free trial of Confluent Cloud today 05

4. Understanding Conﬂuent’s Product Strategy 01

5. Finding your solution for Apache Kafka Weighing the expenses and risks of today’s different deployment options Self Managing Networking spend for increased throughput Excess, underutilized storage/compute Resources to manage infrastructure failures Managed Services Snowflake …comes with unnecessary cost and complexity. …don’t serve all your data streaming needs. …continue to deliver success across every category. Hosted Services Networking considered “out of scope”, and can be 90% of your TCO Kafka reliability, patching, upgrades, etc. on the user Incomplete solutions that don’t address all data streaming needs

6. Conﬂuent Cloud Cloud Native Apache Kafka® , fully managed and re-architected to harness the power of the cloud Complete Go above & beyond Kafka with all the essential tools for a complete data streaming platform Everywhere Connect your data in real time with a platform that spans from on-prem to cloud and across clouds Cloud-native data streaming platform that is 10x better than Kafka Conﬂuent Cloud customers lower their overall TCO for Apache Kafka by up to 60%

7. Simple Hybrid/Multicloud Streaming Confluent powers… Enterprise-grade Security & Governance Modern, cloud-based data systems Cloud-native / SaaS apps Azure Synapse Analytics Legacy data systems Mainframes Applications ksqlDB in-flight stream processing Infinite storage clusters supporting up to 20 GBps overall throughput with a 99.99% Uptime SLA 120+ Pre-Built Source & Sink Connectors (70+ provided fully managed) Hybrid and Multicloud Cluster Linking | Replicator Self-managed software Kubernetes | VMs | Bare Metal Fully managed cloud service AWS | Azure | GCP Flexible DevOps Automation Terraform Provider | Ansible Playbooks Source Connectors Sink Connectors Custom Throughput Limits by App Custom Throughput Limits by App

8. Quarterly Bundled Launches Everywhere Connect your data in real time with a platform that spans from on-prem to cloud and across clouds Complete Go above & beyond Kafka with all the essential tools for a complete data streaming platform Cloud Native Apache Kafka® , fully managed and re-architected to harness the power of the cloud Expanding upon Cloud Native, Complete, and Everywhere product differentiation Net-new capabilities added to each pillar every quarter

9. Recent Product Updates 02

10. Cloud Native Apache Kafka®, fully managed and re-architected to harness the power of the cloud Highlighted “Cloud Native” updates from the past 12 months Dedicated Cluster Expand & Shrink Increased Uptime SLA for Apache Kafka 99.99% Cloud Client Quotas Higher Throughput for Serverless Clusters Independent Network Lifecycle Management Conﬂuent Terraform Provider

11. Confluent Terraform Provider Built with and officially verified by Hashicorp; Available open source on GitHub. Confluent Cloud Infrastructure as Code CI/CD Automation Multicloud data streaming scaled with GitOps integration + = Ongoing automation and continuous monitoring throughout the real-time application lifecycle Resources & data sources supported include: Kafka Clusters Topics Networks Connectors RBAC Service Accounts + more CODE P L A N BUILD TEST R E L E A S E DEPLOY OPERATE MONITOR CI CD Automate and standardize version-controlled infrastructure management across AWS, Azure, and Google Cloud

12. Complete Go above & beyond Kafka with all the essential tools for a complete data streaming platform Highlighted “Complete” updates from the past 12 months 70+ Fully Managed Connectors Enhanced RBAC Centralized Identity Management (OAuth) Monitoring Integrations Stream Processing Recipes Stream Designer Stream Governance Advanced Spotlight features within today’s product demo

13. Stream Designer, the visual builder for streaming data pipelines Easy. Extensible. Extra-fast. ● Visual UI: Visual canvas provides point-and-click interface to build streaming data pipelines in minutes ● Performant: Experience the high performance of Conﬂuent Cloud ● ksqlDB: Visualize ﬁlter, mapping, aggregate, and joins as pre-built pipeline building blocks instead of writing boilerplate code ● Connectors: Leverage 70+ fully managed connectors and SMTs to connect your data sources / sinks

14. Stream Governance Advanced Enterprise-grade governance and data visibility for production workloads Stream Governance is provided fully managed on Conﬂuent Cloud Stream Quality Deliver trusted, high-quality data streams to the business Stream Catalog Increase collaboration and productivity with self-service data discovery Stream Lineage Understand complex data relationships and uncover more insights (Schema Registry & Validation)

15. Stream Governance Advanced Enterprise-grade governance and data visibility for production workloads Stream Governance is provided fully managed on Conﬂuent Cloud Stream Quality Stream Catalog Stream Lineage (Schema Registry & Validation) Point-in-time Lineage Lineage Search Business Metadata GraphQL API 99.95% SLA 20K Schemas 30 Regions

16. Everywhere Recent platform updates Highlighted “Everywhere” updates from the past 12 months Private Service Connect for Google Cloud Supported Regions Expansion Conﬂuent Platform Updates Global Schema Linking One Conﬂuent CLI Zone Selection for Private Networking Spotlight features within today’s product demo Networking expansion for Cluster Linking

17. Cluster Linking: Simple geo-replication New networking options now supported including Transit Gateway and VPC Peered destination clusters. An easier approach with perfectly mirrored and globally consistent topic replication Global Data Sharing Regional Disaster Recovery Cluster Migrations Cost effective, secure, and performant data transport layer across public clouds and regions. Near real-time DR with failover to perfectly mirrored Kafka topics on a separate region or different cloud provider altogether. Prescriptive, low-downtime migrations between cloud providers, across regions, or from another Kafka service to Conﬂuent Cloud.

18. Demo 03 Hendra Sutanto Senior Solutions Engineer

19. Demo Scenario Everything shown during today’s demo will be made available for download within a GitHub repository Data Engineer Develop a new streaming pipeline which combines in-store and online activity to better personalize every customer experience ● Stream Designer will be used to build the pipeline incorporating user clickstreams with in-store shopping activity ● Prepared data should be setup with real-time replication out to global clusters ● Timing is tight, so any data existing on the platform for reuse should be leveraged AcmeCorp Tenure: ~3 months

20. Start building your own pipeline today Download today’s demo repo from GitHub bit.ly/41oIgCc

21. Conﬂuent Roadmap (sneak peek) 04

22. All topics discussed here are confidential and protected by NDA. Confidential information may include, but shall not be limited to, sales or marketing data or plans, product or service information or roadmaps and/or our requirements or preferences. The non-disclosure agreement is not designed for and will not cover the exchange of confidential information in support of joint development or invention disclosure activities. No photos please. NON-DISCLOSURE REMINDER

23. As our roadmap may change in the future, the features referred to herein may change, may not be delivered on time or may not be delivered at all. This information is not a commitment to deliver any functionality and customers should make their purchasing decisions based upon features that are currently available.

24. Increasing the value of real-time data streams Roadmap highlights Expanded In-ﬂight Processing Apache Flink® Strict Data Contracts Data Policies 3rd Party Integrations Simple Data Sharing Stream Sharing Stream Designer Updates More Flexible Integrations Bring Your Own Connector More Cost-effective Infrastructure Enterprise SKU

25. Expanding in-flight processing capabilities Flink Kafka Streams ksqlDB Fully managed on Confluent Cloud Self-managed on Confluent Cloud and Confluent Platform Fully managed on Confluent Cloud, Self-managed on Confluent Platform SQL at GA, Java and Python to follow Java SQL Future Available Now

26. Our fully managed Flink service will be simple, secure, and scalable Simple Secure Scalable • Fully integrated with Conﬂuent’s enterprise- grade identity and access management tooling • Fully integrated with Stream Governance to ensure high data quality and policies are enforced • Audit logging always enabled for compliance • Immediate, automatic, online scaling for elastic scalability • Cloud-native separation of compute, metadata, state, and storage • Seamless management at scale via Terraform provider • Serverless endpoint offering a uniﬁed API for batch and streaming • Comprehensive standard SQL interface with JDBC driver for data querying • Tiered Kafka-backed tables to provide enhanced performance, data durability, and re-use

27. Confluent Cloud offers the largest portfolio of fully managed connectors in the market 70+ fully managed connectors Amazon S3 Amazon Redshift Amazon DynamoDB Google Cloud Spanner AWS Lambda Amazon SQS Amazon Kinesis Azure Service Bus Azure Event Hubs Azure Synapse Analytics Azure Blob Storage Azure Functions Azure Data Lake Google BigTable Access the Confluent Cloud connectors pricing page to see the full list of available fully managed connectors 27 What’s Next Bring Your Own Connector BYOC enables customers to leverage their own connectors with Confluent managing the underlying Connect workers Single Message Transforms Data Output Preview Connect Log Events Roadmap

28. Optimize integrations with data quality rules Define a schema rule to TRANSFORM a field { "schema": "...", "ruleSet": { "domainRules": [ { "name": "populateFullName", "kind": "TRANSFORM", "type": "CEL_FIELD", "expr": "name == 'fullName' && value == null ? message.firstName + ' ' + message.lastName : value" } ] } } Define a schema rule to VALIDATE a field { "schema": "...", "ruleSet": { "domainRules": [ { "name": "checkSsnLength", "kind": "CONSTRAINT", "type": "CEL", "expr": "size(message.ssn) == 9", "onFailure": "DLQ" } ] } } Schema Registry Producer Kafka serializer Kafka deserializer Consumer Confluent Cloud (execute rules) (execute rules)

29. Metadata Integrations Expand data visibility and productivity with 1st class catalog integrations Data Streams Catalog Kafka Cluster ksqlDB Stream & Table Connector Schema Topic … Technical metadata (automated) User-deﬁned metadata Management, search, & discovery Conﬂuent Cloud UI APIs: REST and GraphQL Tags Descriptions Business Metadata Roadmap Real-time metadata syncing Microsoft Purview GA Functionality

30. Start and scale security with ease Client-side field encryption and access control Schema Registry Producer Kafka serializer Kafka deserializer Consumer Kafka 5. Produce message with encrypted fields 8. Consume message with encrypted fields 6. Get schema & policies 7. Return schema & policies Invalid message 2. Get schema & policies 3. Return schema & policies KMS 4. Use master key(s) to encrypt fields 9. Use master key(s) to decrypt fields 1. Schema & Policy defined Protect sensitive data from any other third parties (incl. Confluent) Selectively grant access to sensitive data by specific individual(s) Meet Infosec / compliance requirements quickly Tag-based encryption allows you to simply tag a field, define the policy, and begin streaming encrypted data securely

31. ● Easy collaboration on live data with partners, customers, and vendors ✓ One-click sharing ✓ Trusted and governed ✓ Secure, granular, auditable ● Single, org-wide portal to discover data streams from trusted sources Stream Sharing Simplify inter-enterprise and B2B data sharing Available Now: In Early Access What’s Next: GA Roadmap

32. Stream Designer: Go beyond expertise silos with a low-code solution for building data pipelines ● Round trip: Changes made in the UI are reﬂected in SQL and vice versa ● Future-proof: Easily add or replace data systems or edit stream processing logic without needing to rebuild pipelines ● Troubleshoot: View built-in alerts and pipeline validations for end-to-end testing ● Security: RBAC at the pipeline (Stream Designer) and component (e.g., Kafka, Connect, ksqlDB) levels <SQL> Stream Catalog Integration Dedicated Cluster Support Roadmap Available Now

33. More cost-effective Infrastructure Re-invest in net-new projects when saving with the Enterprise cluster type Basic Get started with scale from $0 pricing For experimentation, early development, and basic use cases. Standard Production-ready for most applications For production- ready use cases leveraging an extended feature set and elastic scaling (up to 250/750MBps) Dedicated Customizable for any application For massive- throughput use cases at GBps+ scale that require private networking Enterprise For use cases with stringent security and networking requirements ● Scale from 0-250/750MBps (ingress/egress) in place automatically ● Private network connectivity: Private Link / Private Services Connect available ● 99.99% Uptime SLA ● Additional premium security features

34. Getting Started 05

35. Deploy how and where you want 35 Conﬂuent AWS MARKETPLACE Azure MARKETPLACE Google Cloud MARKETPLACE Start your multi-cloud journey Simplify procurement with easy click-though through annual commitments and draw down against existing Enterprise Agreements in your cloud marketplace of choice Options for getting started with Conﬂuent Cloud

36. Scalable pricing fit for any budget Start streaming on Confluent Cloud for free and gain industry-leading cost savings Get started quickly with on-demand cluster provisioning and consumption-based billing New signups receive $400 to spend during their first 30 days—no credit card required Access scaled discounts for production or high-volume workloads with committed usage

37. Why Confluent is the world’s most trusted data streaming platform Focus & Expertise Only company focused on data in motion: ● Founded in 2014 by the Original Creators of Apache Kafka ● Over 80% of Kafka commits are by Confluent employees ● Advised on thousands of real-world Kafka deployments across a wide range of patterns & industries Building and supporting a world class product: ● >9 million engineering hours spent building building our product ● We internally manage >30,000 clusters (and counting) in Confluent Cloud ● Over 1 million cumulative hours of Kafka expertise within Confluent support & services Execution at Scale

38. Start your free trial of Confluent Cloud & get $400 in credits Get started with Confluent Cloud is easy! confluent.io/confluent-cloud/tryfree/ Request a personalized TCO assessment for your current Kafka needs or future Data Streaming Platform

39. Our partners include global and regional industry- leading consulting, services and technology companies. Regardless of your size or industry, our partners can help you buy, build, implement, service, support and run Conﬂuent Platform. Conﬂuent Partners Ensure Your Success Featured partners:

40. Q&A

DIMT '23 Session_Demo_ Latest Innovations Breakout.pdf

DIMT '23 Session_Demo_ Latest Innovations Breakout.pdf

Recomendados

Mais conteúdo relacionado

Similar a DIMT '23 Session_Demo_ Latest Innovations Breakout.pdf

Similar a DIMT '23 Session_Demo_ Latest Innovations Breakout.pdf(20)

Mais de confluent

Mais de confluent(20)

Último

Último(20)

DIMT '23 Session_Demo_ Latest Innovations Breakout.pdf