MQTT (Message Queuing Telemetry Transport,) is a message protocol based on the pub/sub model with the advantages of compact message structure, low resource consumption, and high efficiency, which is suitable for IoT applications with low bandwidth and unstable network environments. This session will introduce MQTT on Pulsar, which allows developers users of MQTT transport protocol to use Apache Pulsar. I will share the architecture, principles and future planning of MoP, to help you understand Apache Pulsar's capabilities and practices in the IoT industry.

1. MQTT on Pulsar (MoP)
Jiwei Guo 2022-01-15

2. StreamNative
- Apache Pulsar Committer
- MoP Maintainer
About me
https://github.com/Technoboy-

3. Agenda
- Why MoP?
- Protocol handler
- Support Features
- Implementation details
- Future Work

4. 1. Why MoP?

5. As a MQTT user
•Rewrite the client
•MQTT API wrapper
• MQTT connector

6. With MoP
•No need to change existed MQTT client code, no matter it’s
Java or C++, etc.
•No need to write a new connector for Pulsar.
•MQTT Producer & Pulsar Consumer or Pulsar producer &
MQTT consumer.

7. Proxy/gateway?
MQTT Producer Proxy/Gateway
MQTT
requests
Pulsar Broker
Pulsar
requests Bookie
MoP is a plugin, which means MoP can access all the internal resources of
broker instead sending requests by a client.
MoP
Bookie
BK client
Pulsar Broker
MQTT Producer

8. 2. Protocol handler

9. What’s Protocol Handler?
Protocol handler was introduced
from Pulsar 2.5.0 as a plugin that is
loaded dynamically by broker.

10. Definition of the protocol handler

11. Load the protocol handler
Create an yml
f
ile under META-INF/services/, building the project to NAR package
name: mqtt
description: MQTT Protocol Handler
handlerClass: io.streamnative.pulsar.handlers.mqtt.MQTTProtocolHandler
When a broker starts, it will
- Load all NAR packages under protocolHandlerDirectory
- Select the package whose name is in messagingProtocols
- Initialized the Java class speci
f
ied in handlerClass
- Verify if the name could be accepted by ProtocolHandler#accept.

12. Start protocol handler

13. Install MoP
• Compile -> package -> generate *.nar package.
• Add some extra con
f
igs to broker.conf or standalone.conf.
•
• Restart Pulsar broker

14. MoP Proxy
Extend MoP to multiple nodes to achieve horizontally scalable services
1. MQTT clients connect to MoP Proxy.
2. MoP Proxy sends lookup topic to pulsar cluster, searching for topic owner broker URL。
3. Pulsar cluster gives back topic owner broker URL to MoP Proxy。
4. MoP Proxy connects to owner broker，MQTT Clients begin to communicate with owner broker.

15. 3. Support features

16. - Support MQTT3, MQTT3.1, MQTT5
- Qos0, Qos1
- Authentication
- Authorization
- TLS
- Topic Filter
- Metrics
Features

17. Authentication
mqttAuthenticationEnabled=true
mqttAuthenticationMethods=basic
authenticationProviders=
org.apache.pulsar.broker.authentication.AuthenticationProviderBasic
System.setProperty("pulsar.auth.basic.conf", “/xxx/basic”);
MQTT mqtt = createMQTTClient();
mqtt.setUserName("xxx");
mqtt.setPassword(“pwd");
mqttAuthenticationEnabled=true
mqttAuthenticationMethods=token
authenticationProviders=
org.apache.pulsar.broker.authentication.AuthenticationProviderToken
tokenSecretKey=data:;base64,hXCl3lAdPA=
MQTT mqtt = createMQTTClient();
mqtt.setPassword(“hXCl3lAdPA=");
- Basic
- Token

18. Authorization
mqttAuthorizationEnabled=true
MQTT mqtt = createMQTTClient();
mqtt.setUserName("admin10");
mqtt.setPassword(“123”);

19. TLS
-TLS
tlsEnabled=true
mqttListeners=mqtt+ssl://127.0.0.1:8883
tlsCertificateFilePath=/xxx/server.crt
tlsKeyFilePath=/xxx/server.key
-TLS-PSK
tlsPskEnabled=true
mqttListeners=mqtt+ssl+psk://127.0.0.1:8884
tlsPskIdentityHint=alpha
tlsPskIdentity=mqtt:mqtt123

20. Topic Filter
/a/b/c
/a/b/# multi-level wildcard
/a/+/c single level wildcard
Tips:
-Topic filter could not listener for new topics.

21. Metrics
-Prometheus
-Http endpoint
additionalServlets=mqtt-servlet
additionalServletDirectory=[protocolHandlerDir]
curl http://pulsar-broker-webservice-address:port/mop-stats/
{“cluster":"test","subscriptions":{"subs":["/a/b/c"],"count":1},"clients":
{"total":1,"maximum":1,"active":0,"active_clients":[]},"namespace":"default","messages":
{"received_bytes":57351,"received_count":10,"send_count":20,"send_bytes":60235},"ver
sion":"2.9.0-SNAPSHOT","tenant":"public","uptime":"46 seconds"}
MoP will uniformly output its own metrics to Prometheus.

22. 4. Implementation details

23. Topic map
=> persistent://public/default/topicA
- tenantA/NamespaceA/topicA => persistent://tenantA/NamespaceA/topicA
- /a/b/c => persistent://public/default/%2Fa%2Fb%2Fc
- topicA
- persistent://public/default/a/b
- non-persistent://public/default/a/b

24. Connect

25. Publish

26. Subscribe

27. Unsubscribe

28. Disconnect

29. 5. Future Work

30. - Performance Test
- Support Qos2
- Management API

31. THANK YOU