Kafka Connect makes it possible to easily integrate data sources like MongoDB! In this session we will first explore how MongoDB enables developers to rapidly innovate through the use of the document model. We will then put the document model to life and showcase how to integrate MongoDB and Kafka through the use of the MongoDB Connector with Apache Kafka. Finally, we will explore the different ways of using the connector including the new Confluent Cloud integration.

1. Making your life easier
with MongoDB and
Kafka
Robert Walters | MongoDB

2. Presenter
Robert Walters
Product Manager
Connectors and Things
MongoDB
https://www.linkedin.com/in/robwaltersprofile/

3. Agenda
MongoDB Primer
Deep Dive: MongoDB Kafka Connector
Demo

4. MongoDB Adoption
90,000,000+
MongoDB Downloads
1 500,000+
Online Education Students
1m+
MongoDB Atlas Clusters
75,000+
MongoDB User Group
Members
1,000+
Technology and Services
Partners
17,500+
Customers Across All
Industries
DB-Engines Rankings
Fastest Growing Database over
the past decade
Worldwide Activations

5. How does MongoDB make my
life easier?

6. Strict Schema vs. Flexible Data Model
id title description
12345-WoTLamp-1234 My Lamp A lamp in the room
67890-WoTLamp-1234 My Other Lamp Another lamp in the room
id thing_id title readOnly writeOnly
4711-p1 12345-WoTLamp-1234 status false false
id thing_id title safe idempodent
4711-a1 12345-WoTLamp-1234 toggle false false
id thing_id title readOnly writeOnly
4711-e1 12345-WoTLamp-1234 overheating false false
Thing
Property
Action
Event
{
"id": "123456-WoTLamp-1234",
"title": "My Lamp",
"description": "A lamp in the room",
"properties": {
"status": {
"type": "string",
"readOnly" : false,
"writeOnly" : false
}
},
"actions": {
"toggle": {
"safe": false,
"idempodent": false
}
},
"events": {
"overheating": {
"data": {
"type": "string",
"readOnly" : false,
"writeOnly" : false
}
}
}
}
Relational Example Flexible Document Model

7. Telediagnostics: The Future of Mercedes Benz Services
Learn more about the project in Madalin Broscaru’s MongoDB.live presentation:
Telediagnostics@Mercedes Benz powered by MongoDB
https://www.youtube.com/watch?v=MEH9_P_NATk&list=PL4RCxklHWZ9uRUB9qTnZE9cauOsTGE22n&index=50
… the Vehicle Data
Conditioning (VDC) where
these technical vehicle events
are processed …
CAC
Retail
Customer
… and the follow-up processes are
triggered with real-time
recommendations for actions.
Vehicles are transmitting regularly
status and health data into ...
Aggregated
Quality Analysis

8. Telediagnostics: The Future of Mercedes Benz Services
{
.............
"schema": "3.1.0"
"createdAt": {..},
.............
"vehicleIdentData": {
"chassisNumber": "WDD24708A5432J63",
"countryCode": "4f3490b14e238a5f",
"modelSeries": "f16ad22d42064811",
"modelType": "2196868af1c70d74",
"modelYear": "5e01ac15d73c3e4a",
"steering": "4a3424fe6411461c"
},
"basicData": {
"mileage": {..},
"batteries": [..],
"tanks": [..],
"tiresPressure": [..],
},
"controlUnits": [..],
"affectedFunctions": [..],
"vehicleClusterMessagesData": [..],
"maintenanceData": {...},
..............
}
"controlUnits": [
{
"ecuId": {..}
"name": "a927e49b0549f00f71",
"detailsHardware": {},
"detailsSoftware": {},
"dtc": [
{
"code": "B214F73",
...
"failureText": "81957650ea",
"environmentalData": {...}
},
...
]
},
...
]
More presentations available:
https://www.mongodb.com/world

9. Self-hosted MongoDB Turn-key modern database
You install, patch,
maintain, scale, etc..
MongoDB as a service plus
a whole lot more…
MongoDB & MongoDB Atlas

10. MongoDB Atlas demo

11. Kafka Connect
Connect Connect
https://hub.confluent.io
[sources
]
[sinks]
ANY e.g. file systems, data stores, REST endpoints, …
Kafka Connect is an easy way to get data from many heterogeneous platforms into and out of Kafka

12. Available on the Confluent Hub:
https://www.confluent.io/hub/mongodb/kafka-connect-mongodb
MongoDB Connector for Apache Kafka
Supported by MongoDB Verified Gold by

13. #MDBLocal
MongoDB Source
MongoDB Database
MongoDB
Connector
topicA
topicB
topicC
Kafka Cluster
Subscribes to
Change Stream
events
Writes to Kafka
Topic(s)
When MongoDB is a Kafka Source...
Subscribes to change stream events and writes to Kafka topic(s)

14. #MDBLocal
MongoDB Source: Writing to a Topic
Writes to topic based on database and collection name
Be aware of your message sizes
Optionally specify pipeline to manage change stream output to watch
Optionally set a topic.prefix in the connector configuration
database
collection
{}
[topic.prefix].database-name.collection-name
writes to topic ->
connector
pipeline[]
change
stream

15. #MDBLocal
MongoDB Sink
MongoDB Database
MongoDB
Connector
topicA
topicB
topicC
Kafka Cluster
Writes
documents to db
collection
Reads from
Kafka Topic(s)
When MongoDB is a Kafka Sink...
Reads from Kafka topic(s) and writes documents to collection

16. #MDBLocal
MongoDB Sink: Reading Messages from Topic
Reads messages from topic (based on pointer to message in topic)
Writes message into MongoDB database collection
Moves pointer to next message based on write to database
Kafka Topic
connector
database
collection
{}
1: pointer to
message to read
3: on successful write,
moves pointer to next

17. MongoDB Connector
Configuring the connector as a source and sink
Aging data out to cost-effective cloud storage with Atlas Online Archive

18. MongoDB Atlas Connector in
Confluent Cloud!

19. Two Ways to Stream
1919
SELF-MANAGED SOFTWARE
Confluent Platform
The Enterprise Distribution of Apache Kafka
Deploy on any platform, on-prem or cloud
VM
FULLY-MANAGED SOFTWARE
Confluent Cloud
Apache Kafka Re-Engineered for the Cloud
Available on the leading public clouds
Option to purchase via GCP marketplace
Both: Subscription products where price scales with usage

20. Confluent Cloud MongoDB Atlas Connector

21. Preview Limitations
Source
• MongoDB Atlas only.
• Public access (0.0.0.0/0) must be
allowed by Atlas.
• Output of this connector is String.
• 1 task / connector limitation
• Pipeline parameter not available
Sink
● MongoDB Atlas only.
● The MongoDB Atlas and Kafka cluster
must be in the same region.
● Public access (0.0.0.0/0) must be allowed
by Atlas.
● 1 task / connector limitation
NOTE: These preview limitations only apply to the MongoDB Atlas
connector in the Confluent Cloud. The MongoDB Connector for Apache
Kafka can be used in a self-hosted environment with MongoDB Atlas
without these network restrictions.

22. Using MongoDB Atlas with
Confluent Cloud

23. Want to Know More?
https://www.confluent.io/resources/confluent-platform-reference-architecture-mongodb/

24. Making your life easier
Wrap Up
Document Model
and MQL
The fastest way
to innovate
MongoDB Connector for
Apache Kafka
Gold Certified - Available
on Confluent Cloud!
MongoDB
Atlas:
More than a database -
a data platform

25. Thank you
That’s all folks
Robert Walters | Robert.Walters@MongoDB.com | MongoDB