Kafka Security

© Hortonworks Inc. 2014
Kafka Security
SSL, Kerberos & Authorization

Who Are We?
Sriharsha Chintalapani
Apache Kafka Committer
Apache Storm Committer & PMC
Parth Brahmbhatt
Apache Kafka Contributor
Apache Storm Committer & PMC

Why Kafka Security?
• Kafka is becoming centralized data bus connecting
external data sources to Hadoop eco system.
• There are lot of requests/discussions in Kafka mailing
lists to add security

Why Kafka Security?
• How can we prevent rogue agents to
publishing/consuming data from Kafka
• How can we encrypt the data that’s flowing through the
network
• How can we give permissions to a topic to specific
group or users

Kafka Security
• We recognized the necessity of security in Kafka
• Added wire encryption via SSL
• Role Based authentication via SASL ( Kerberos)
• Authorizer to add fine-grain access controls to Kafka
topics per User, per Host.

Kafka Networking

Kafka Networking
http://www.slideshare.net/jjkoshy/troubleshooting-kafkas-socket-server-from-incident-to-resolution

SSL

Kafka Security – SSL
• Kafka SSL / SASL requirements
• No User-level API changes to clients
• Retain length-encoded Kafka protocols
• Client must authenticate before sending/receiving requests
• Kafka Channel
• Instead of using socket channel, we added KafkaChannel
which consists a TransportLayer, Authenticator.

• SSLTransportLayer
• Before sending any application data, both client and server
needs to go though SSL handshake
• SSLTransportLayer uses SSLEngine to establish a non-
blocking handshake.
• SSLEngine provides a state machine to go through several
steps of SSLhandshake

Kafka Networking
KafkaChannel
TransportLayer
Authenticator
Kafka Server
handshake
authenticate

• SSLTransportLayer
• SocketChannel read
• Returns encrypted data
• Decrypts the data and returns the length of the data from Kafka protocols
• SocketChannel Write
• Writes encrypted data onto channel
• Regular socketChannel returns length of the data written to socket.
• Incase of SSL since we encrypt the data we can’t return exact length written to
socket which will be more than actual data
• Its important to keep track length of data written to network. This signifies if we
successfully written data to the network or not and move on to next request.

• Principal Builder
• SSLTransportLayer gives hostname as authenticated user
• X509Certificate has lot more information about a client
identity.
• PrincipalBuilder provides interface to plug in a custom
PrincipalBuilder that has access to X509Certificate and can
construct a user identity out of it.
• Authenticator can use this custom principal to add ACLs

• listeners=SSL://host.name:port
• ssl.keystore.location
• ssl.keystore.password
• ssl.key.password
• ssl.truststore.location
• ssl.truststore.password
• security.inter.broker.protocol (optional)

SASL/Kerberos

Kafka Security – SASL
• Simple Authentication and Security Layer, or SASL
• Provides flexibility in using Login Mechanisms
• One can use Kerberos , LDAP or simple passwords to authenticate.
• JAAS Login
• Before client & server can handshake , they need to authenticate with
Kerberos or other Identity Provider.
• JAAS provides a pluggable way of providing user credentials. One can
easily add LDAP or other mechanism just by changing a config file.

• Pass JAAS config file as jvm parameter. -
Djava.security.auth.login.config
• JAAS Config file
KafkaServer {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
storeKey=true
serviceName="kafka"
keyTab="/vagrant/keytabs/kafka1.keytab"
principal="kafka/host@EXAMPLE.COM";
};
KafkaClient {
useKeyTab=true
storeKey=true
serviceName="kafka"
keyTab="/vagrant/keytabs/client1.keytab"
principal=”client/host@EXAMPLE.COM";
};

Client Broker
Connection
Mechanism list
Selected Mechanism & sasl data
Evaluate and Response
Sasl data
Client Authenticated

Kafka Security – Resources
• SSL
• https://cwiki.apache.org/confluence/display/KAFKA/Deploying+SSL+for+Kafka
• SASL
• https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61326390
• Vagrant Setup
• SASL
• https://github.com/harshach/kafka-vagrant/tree/master/
• SSL
• https://github.com/harshach/kafka-vagrant/tree/ssl/

Authorization

Authorizer
• Controls who can do what
• Pluggable
• Acl based approach

Acl
• Alice is Allowed to Read from Orders-topic from Host-1
Principal Permission Operation Resource Host
Alice Allow Read Orders Host-1

Principal
• PrincipalType:Name
• Supported types: User
• Extensible so users can add their own types
• Wild Card User:*

Operation
• Read, Write, Create, Delete, Alter, Describe,
ClusterAction, All
• Each API as an Operation VS Classification that maps to
APIs.

Resource
• ResourceType:ResourceName
• Topic, Cluster and ConsumerGroup
• Wild card resource ResourceType:*

Permissions
• Allow and Deny
• Anyone without an explicit Allow ACL is denied
• Then why do we have Deny?
• Deny works as negation
• Deny takes precedence over Allow Acls

Hosts
• Why provide this granularity?
• Allows authorizer to provide firewall type security even in
non secure environment.
• * as Wild card.

Configuration
• Authorizer class
• Super users
• Authorizer properties
• Default behavior for resources with no ACLs

SimpleAclAuthorizer
• Out of box authorizer implementation.
• Stores all of its ACLs in zookeeper.
• In built ACL cache to avoid performance penalty.
• Provides authorizer audit log.

Client Broker Authorizer Zookeeper
configure
Read ACLs
Load
Cache
Request
authorize
ACL match
Or Super User?
Allowed/Den
ied

CLI
• Add, Remove and List acls
• Convenience options:
--producer and --consumer.

Ranger Policy

Ranger Auditing

Ranger ACL management Audit

Unsecure zookeeper

Zookeeper
• Kafka’s metadata store
• Has its own security mechanism that supports SASL and
MD5-DIGEST for establishing identity and ACL based
authorization
• Create , Delete directly interacts with zookeeper

Securing zookeeper
• Acl on zk nodes: user:cdrwa
• Zookeeper.set.acl
• ZkSecurityMigrator script
• Credit where its due: Flavio Junqueira

Client JAAS
Client {
useKeyTab=true
storeKey=true
serviceName="zookeeper"
keyTab="/vagrant/keytabs/kafka.keytab"
principal="kafka/kafka@WITZEND.COM";
};

Future
• KIP-4: Move everything to server side, no direct
interactions with zookeeper
• Group Support
• Pluggable Auditor
• Delegation Tokens
• Impersonation

Summary
• SSL for wire encryption
• Sasl for authentication
• Authorization
• Secure Zookeeper
Thanks to the community for participation.

Kafka Security

Recomendados

Recomendados

Mais conteúdo relacionado

Mais procurados

Mais procurados (20)

Destaque

Destaque (20)

Semelhante a Kafka Security

Semelhante a Kafka Security (20)

Mais de DataWorks Summit/Hadoop Summit

Mais de DataWorks Summit/Hadoop Summit (20)

Último

Último (20)

Kafka Security