Watch this talk here: https://www.confluent.io/online-talks/gcp-for-apache-kafka-users-stream-ingestion-processing In private and public clouds, stream analytics commonly means stateless processing systems organized around Apache Kafka® or a similar distributed log service. GCP took a somewhat different tack, with Cloud Pub/Sub, Dataflow, and BigQuery, distributing the responsibility for processing among ingestion, processing and database technologies. We compare the two approaches to data integration and show how Dataflow allows you to join and transform and deliver data streams among on-prem and cloud Apache Kafka clusters, Cloud Pub/Sub topics and a variety of databases. The session will have a mix of architectural discussions and practical code reviews of Dataflow-based pipelines.

1. GCP for Apache
Kafka® Users:
Stream Ingestion
and Processing
Karthi Thyagarajan, Solutions Architect, Google Cloud
Ricardo Ferreira, Developer Advocate, Confluent (@riferrei)

2. Speakers
Karthi Thyagarajan
Solutions Architect, Google Cloud
Ricardo Ferreira
Developer Advocate, Confluent

3. Confluent Wins Google Cloud Technology Partner
of the Year Award for Data and Analytics

4. GCP’s architecture for Stream Analytics
Endpoint
clients
User &
device data
Or Or
Ingest Transform Analyze
(data warehouse)
Web
IoT
Mobile
PubSub
Apache
Kafka
Apache
Beam
Dataflow
KSQL
BigQuery
ML
BigTable
Data
Studio
3rd-party
BI Tools
Data
consumers

5. Components
Ingestion & delivery
Df
KS
S
PS
K
Df
KC
S
Cloud Pub/Sub
Apache Kafka
Processing
Apache Beam / Dataflow
Kafka Streams / KSQL
Apache Spark / Dataproc
Connectors
Apache Beam IOs
Kafka Connect
Apache Spark / Dataproc

6. What is Beam and Dataflow?
Open source programming model
Unified batch and streaming
Exactly-once streaming semantics
Cloud
Dataflow
Uses Apache Beam as its SDK
Automatic optimizations scale to millions of QPS
Serverless, fully managed data processing

7. Apache Kafka as a Service for Google Cloud

8. Apache Kafka as a Service for Google Cloud
Clients Connectors Management
More than simply a managed cluster service!

9. Demo: Creating a
Kafka Cluster in
GCP in Less Than
3 Minutes using

10. Demo Processing

11. Demo: Join Kafka & Pub/Sub Topics
On-Premise
Cloud
Dataflow
Orange arrow
indicates
Cloud
Dataflow
PS
BQ
Cloud App
CCE cluster in a
peered VPC
On-Prem
App DB
KC

12. Credit
Applications
{
"ApplicationId": "997c7045-3497-4aba-b7fe-8ff48d4e8dd3",
"FirstName": "Felipe",
"LastName": "Jast",
"CreditScore": 644,
"IPAddress": "64.73.63.94",
"Company": “Company-Jast”,
"EmailAddress": "Felipe.Jast@Company-Jast.com",
"TimestampsAsJSON": "["2018-09-24T17:57:58.2971524Z"]"
}

13. Demo: Showing how to
process stream of events
using Schema Registry
and KSQL
Ricardo

14. Demo take action

15. Demo: Showing how to send
the results of a processing
stream to Cloud Functions
using Kafka Connect

16. Kafka Best Practices for Hybrid Cloud / Cloud Native
Hybrid Cloud Cloud-Native
On-Premises
Network
Applications Network (VPC)
Producer Consumer
Producer Consumer
Replicator
Kafka Producer Kafka Consumer
VPC
Peering
Kafka Cluster Network (VPC)

17. Confluent is Available Everywhere!
BigQuery
Colocated DC
Legacy app
Public Cloud
Another app
Messaging
Apache Kafka
Kafka tools, skills, replication infrastructure
Messaging
Apache Kafka
Messaging
Apache Kafka

18. Try Apache Kafka as a Service for GCP Today!
https://cnfl.io/next19

19. Code: KS19Online25
25% DISCOUNT*
*Standard Priced Conference pass

20. Thank you