SlideShare a Scribd company logo
1 of 15
Download to read offline
Fluentd message forwarding
                       with authentication
                           and encryption

                        @tagomoris (TAGOMORI Satoshi)
                   Fluentd Casual Talks #2 at 2013/02/15



13年2月15日金曜日
TAGOMORI SATOSHI (@TAGOMORIS)
                    NHN JAPAN CORP.



13年2月15日金曜日
13年2月15日金曜日
Fluentd forward
              has no authentication

                issues:81 "Authentication between nodes is
                required" by @saki7

              has no encryption

                pull:56 "Added compression and encryption
                feature to forward plugin" by @mazgi



13年2月15日金曜日
But how?

         'OpenSSL::SSL::SSLSocket'
              Socket overwrapped by OpenSSL
         Simple digest authentication over SSL



13年2月15日金曜日
fluent-plugin-secure-forward

              Forwarding over SSL [required]

                with auto-generated self-signed certification

              Shared key authentication [required]

                to verify receiver identity

              Username and Password authentication [optional]

                to verify sender identity


13年2月15日金曜日
Connect:
         out_secure_forward


                    1. CONNECT TCP SESSION
                    2. START SSL SESSION
                    3. CHECK CERTIFICATE TTL
                    (4. CHECK CERTIFICATE CN)

          in_secure_forward



13年2月15日金曜日
Handshake (1):
         out_secure_forward


                    1. SERVER SEND HELO
                     - SERVER OPTIONS
                     - AUTH_SALT




          in_secure_forward



13年2月15日金曜日
Handshake (2):
         out_secure_forward

                    2. CLIENT SEND PING
                     - CLIENT HOST NAME
                     - SALT FOR SHARED KEY
                     - DIGEST(SALT + HOSTNAME + SHARED KEY)
                     - USERNAME
                     - DIGEST(AUTH_SALT + USERNAME + PASSWORD)

          in_secure_forward



13年2月15日金曜日
Handshake (3):
         out_secure_forward


                   3. SERVER CHECK PING
                     - SOURCE IP/HOST PERMISSION
                     - SHARED KEY DIGEST
                     - USERNAME AND PASSWORD DIGEST


          in_secure_forward



13年2月15日金曜日
Handshake (4):
         out_secure_forward


                    4. SERVER SEND PONG
                     - AUTHENTICATION RESULT
                     - SERVER HOST NAME
                     - DIGEST(SALT + HOSTNAME + SHARED KEY)



          in_secure_forward



13年2月15日金曜日
Handshake (5):
         out_secure_forward


                    5. CLIENT CHECK PONG
                     - AUTHENTICATION RESULT
                     - SHARED KEY DIGEST




          in_secure_forward



13年2月15日金曜日
Session established:
         out_secure_forward


                    * SEND MESSAGES OVER SSL




          in_secure_forward



13年2月15日金曜日
Many many TODOs:
              Tests for non self-signed certificates
              ACK mode protocol design & implementation
              Keepalive disable mode implementation
              More access control
                Network ACL & domain wildcard ACL
                DNS reverse lookup & check
                Disconnection just after SSL session established
              Pluggable authentication databases
              Encryption algorithm options
              Balancing/failover connections
              TESTS!!!!!!! (this plugin has no one asserts....)


13年2月15日金曜日
fluent-plugin-secure-forward
              Version v0.0.1 is HIGHLY EXPERIMENTAL

              This plugin is TOY PROGRAM PoC

              We needs other maintainer who uses this plugin

                or MONEY & TD support! :-)


                                               Thanks!

13年2月15日金曜日

More Related Content

More from SATOSHI TAGOMORI

Data Processing and Ruby in the World
Data Processing and Ruby in the WorldData Processing and Ruby in the World
Data Processing and Ruby in the WorldSATOSHI TAGOMORI
 
Planet-scale Data Ingestion Pipeline: Bigdam
Planet-scale Data Ingestion Pipeline: BigdamPlanet-scale Data Ingestion Pipeline: Bigdam
Planet-scale Data Ingestion Pipeline: BigdamSATOSHI TAGOMORI
 
Technologies, Data Analytics Service and Enterprise Business
Technologies, Data Analytics Service and Enterprise BusinessTechnologies, Data Analytics Service and Enterprise Business
Technologies, Data Analytics Service and Enterprise BusinessSATOSHI TAGOMORI
 
Ruby and Distributed Storage Systems
Ruby and Distributed Storage SystemsRuby and Distributed Storage Systems
Ruby and Distributed Storage SystemsSATOSHI TAGOMORI
 
Perfect Norikra 2nd Season
Perfect Norikra 2nd SeasonPerfect Norikra 2nd Season
Perfect Norikra 2nd SeasonSATOSHI TAGOMORI
 
To Have Own Data Analytics Platform, Or NOT To
To Have Own Data Analytics Platform, Or NOT ToTo Have Own Data Analytics Platform, Or NOT To
To Have Own Data Analytics Platform, Or NOT ToSATOSHI TAGOMORI
 
The Patterns of Distributed Logging and Containers
The Patterns of Distributed Logging and ContainersThe Patterns of Distributed Logging and Containers
The Patterns of Distributed Logging and ContainersSATOSHI TAGOMORI
 
How To Write Middleware In Ruby
How To Write Middleware In RubyHow To Write Middleware In Ruby
How To Write Middleware In RubySATOSHI TAGOMORI
 
Modern Black Mages Fighting in the Real World
Modern Black Mages Fighting in the Real WorldModern Black Mages Fighting in the Real World
Modern Black Mages Fighting in the Real WorldSATOSHI TAGOMORI
 
Open Source Software, Distributed Systems, Database as a Cloud Service
Open Source Software, Distributed Systems, Database as a Cloud ServiceOpen Source Software, Distributed Systems, Database as a Cloud Service
Open Source Software, Distributed Systems, Database as a Cloud ServiceSATOSHI TAGOMORI
 
Fluentd Overview, Now and Then
Fluentd Overview, Now and ThenFluentd Overview, Now and Then
Fluentd Overview, Now and ThenSATOSHI TAGOMORI
 
How to Make Norikra Perfect
How to Make Norikra PerfectHow to Make Norikra Perfect
How to Make Norikra PerfectSATOSHI TAGOMORI
 
Distributed Logging Architecture in Container Era
Distributed Logging Architecture in Container EraDistributed Logging Architecture in Container Era
Distributed Logging Architecture in Container EraSATOSHI TAGOMORI
 
Fighting API Compatibility On Fluentd Using "Black Magic"
Fighting API Compatibility On Fluentd Using "Black Magic"Fighting API Compatibility On Fluentd Using "Black Magic"
Fighting API Compatibility On Fluentd Using "Black Magic"SATOSHI TAGOMORI
 
Fluentd v0.14 Plugin API Details
Fluentd v0.14 Plugin API DetailsFluentd v0.14 Plugin API Details
Fluentd v0.14 Plugin API DetailsSATOSHI TAGOMORI
 
Overview of data analytics service: Treasure Data Service
Overview of data analytics service: Treasure Data ServiceOverview of data analytics service: Treasure Data Service
Overview of data analytics service: Treasure Data ServiceSATOSHI TAGOMORI
 
Hive dirty/beautiful hacks in TD
Hive dirty/beautiful hacks in TDHive dirty/beautiful hacks in TD
Hive dirty/beautiful hacks in TDSATOSHI TAGOMORI
 
Data Analytics Service Company and Its Ruby Usage
Data Analytics Service Company and Its Ruby UsageData Analytics Service Company and Its Ruby Usage
Data Analytics Service Company and Its Ruby UsageSATOSHI TAGOMORI
 
Tale of ISUCON and Its Bench Tools
Tale of ISUCON and Its Bench ToolsTale of ISUCON and Its Bench Tools
Tale of ISUCON and Its Bench ToolsSATOSHI TAGOMORI
 

More from SATOSHI TAGOMORI (20)

Data Processing and Ruby in the World
Data Processing and Ruby in the WorldData Processing and Ruby in the World
Data Processing and Ruby in the World
 
Planet-scale Data Ingestion Pipeline: Bigdam
Planet-scale Data Ingestion Pipeline: BigdamPlanet-scale Data Ingestion Pipeline: Bigdam
Planet-scale Data Ingestion Pipeline: Bigdam
 
Technologies, Data Analytics Service and Enterprise Business
Technologies, Data Analytics Service and Enterprise BusinessTechnologies, Data Analytics Service and Enterprise Business
Technologies, Data Analytics Service and Enterprise Business
 
Ruby and Distributed Storage Systems
Ruby and Distributed Storage SystemsRuby and Distributed Storage Systems
Ruby and Distributed Storage Systems
 
Perfect Norikra 2nd Season
Perfect Norikra 2nd SeasonPerfect Norikra 2nd Season
Perfect Norikra 2nd Season
 
Fluentd 101
Fluentd 101Fluentd 101
Fluentd 101
 
To Have Own Data Analytics Platform, Or NOT To
To Have Own Data Analytics Platform, Or NOT ToTo Have Own Data Analytics Platform, Or NOT To
To Have Own Data Analytics Platform, Or NOT To
 
The Patterns of Distributed Logging and Containers
The Patterns of Distributed Logging and ContainersThe Patterns of Distributed Logging and Containers
The Patterns of Distributed Logging and Containers
 
How To Write Middleware In Ruby
How To Write Middleware In RubyHow To Write Middleware In Ruby
How To Write Middleware In Ruby
 
Modern Black Mages Fighting in the Real World
Modern Black Mages Fighting in the Real WorldModern Black Mages Fighting in the Real World
Modern Black Mages Fighting in the Real World
 
Open Source Software, Distributed Systems, Database as a Cloud Service
Open Source Software, Distributed Systems, Database as a Cloud ServiceOpen Source Software, Distributed Systems, Database as a Cloud Service
Open Source Software, Distributed Systems, Database as a Cloud Service
 
Fluentd Overview, Now and Then
Fluentd Overview, Now and ThenFluentd Overview, Now and Then
Fluentd Overview, Now and Then
 
How to Make Norikra Perfect
How to Make Norikra PerfectHow to Make Norikra Perfect
How to Make Norikra Perfect
 
Distributed Logging Architecture in Container Era
Distributed Logging Architecture in Container EraDistributed Logging Architecture in Container Era
Distributed Logging Architecture in Container Era
 
Fighting API Compatibility On Fluentd Using "Black Magic"
Fighting API Compatibility On Fluentd Using "Black Magic"Fighting API Compatibility On Fluentd Using "Black Magic"
Fighting API Compatibility On Fluentd Using "Black Magic"
 
Fluentd v0.14 Plugin API Details
Fluentd v0.14 Plugin API DetailsFluentd v0.14 Plugin API Details
Fluentd v0.14 Plugin API Details
 
Overview of data analytics service: Treasure Data Service
Overview of data analytics service: Treasure Data ServiceOverview of data analytics service: Treasure Data Service
Overview of data analytics service: Treasure Data Service
 
Hive dirty/beautiful hacks in TD
Hive dirty/beautiful hacks in TDHive dirty/beautiful hacks in TD
Hive dirty/beautiful hacks in TD
 
Data Analytics Service Company and Its Ruby Usage
Data Analytics Service Company and Its Ruby UsageData Analytics Service Company and Its Ruby Usage
Data Analytics Service Company and Its Ruby Usage
 
Tale of ISUCON and Its Bench Tools
Tale of ISUCON and Its Bench ToolsTale of ISUCON and Its Bench Tools
Tale of ISUCON and Its Bench Tools
 

Fluentd message forwarding with authentication and encryption

  • 1. Fluentd message forwarding with authentication and encryption @tagomoris (TAGOMORI Satoshi) Fluentd Casual Talks #2 at 2013/02/15 13年2月15日金曜日
  • 2. TAGOMORI SATOSHI (@TAGOMORIS) NHN JAPAN CORP. 13年2月15日金曜日
  • 4. Fluentd forward has no authentication issues:81 "Authentication between nodes is required" by @saki7 has no encryption pull:56 "Added compression and encryption feature to forward plugin" by @mazgi 13年2月15日金曜日
  • 5. But how? 'OpenSSL::SSL::SSLSocket' Socket overwrapped by OpenSSL Simple digest authentication over SSL 13年2月15日金曜日
  • 6. fluent-plugin-secure-forward Forwarding over SSL [required] with auto-generated self-signed certification Shared key authentication [required] to verify receiver identity Username and Password authentication [optional] to verify sender identity 13年2月15日金曜日
  • 7. Connect: out_secure_forward 1. CONNECT TCP SESSION 2. START SSL SESSION 3. CHECK CERTIFICATE TTL (4. CHECK CERTIFICATE CN) in_secure_forward 13年2月15日金曜日
  • 8. Handshake (1): out_secure_forward 1. SERVER SEND HELO - SERVER OPTIONS - AUTH_SALT in_secure_forward 13年2月15日金曜日
  • 9. Handshake (2): out_secure_forward 2. CLIENT SEND PING - CLIENT HOST NAME - SALT FOR SHARED KEY - DIGEST(SALT + HOSTNAME + SHARED KEY) - USERNAME - DIGEST(AUTH_SALT + USERNAME + PASSWORD) in_secure_forward 13年2月15日金曜日
  • 10. Handshake (3): out_secure_forward 3. SERVER CHECK PING - SOURCE IP/HOST PERMISSION - SHARED KEY DIGEST - USERNAME AND PASSWORD DIGEST in_secure_forward 13年2月15日金曜日
  • 11. Handshake (4): out_secure_forward 4. SERVER SEND PONG - AUTHENTICATION RESULT - SERVER HOST NAME - DIGEST(SALT + HOSTNAME + SHARED KEY) in_secure_forward 13年2月15日金曜日
  • 12. Handshake (5): out_secure_forward 5. CLIENT CHECK PONG - AUTHENTICATION RESULT - SHARED KEY DIGEST in_secure_forward 13年2月15日金曜日
  • 13. Session established: out_secure_forward * SEND MESSAGES OVER SSL in_secure_forward 13年2月15日金曜日
  • 14. Many many TODOs: Tests for non self-signed certificates ACK mode protocol design & implementation Keepalive disable mode implementation More access control Network ACL & domain wildcard ACL DNS reverse lookup & check Disconnection just after SSL session established Pluggable authentication databases Encryption algorithm options Balancing/failover connections TESTS!!!!!!! (this plugin has no one asserts....) 13年2月15日金曜日
  • 15. fluent-plugin-secure-forward Version v0.0.1 is HIGHLY EXPERIMENTAL This plugin is TOY PROGRAM PoC We needs other maintainer who uses this plugin or MONEY & TD support! :-) Thanks! 13年2月15日金曜日