O slideshow foi denunciado.
Utilizamos seu perfil e dados de atividades no LinkedIn para personalizar e exibir anúncios mais relevantes. Altere suas preferências de anúncios quando desejar.

Fluentd Meetup 2016 - ServerEngine Integration & Windows support

5.545 visualizações

Publicada em

In Fluentd new version v0.14, ServerEngine is integrated.
I'll explain about the advantages of ServerEngine.

Publicada em: Software
  • Seja o primeiro a comentar

Fluentd Meetup 2016 - ServerEngine Integration & Windows support

  1. 1. ServerEngine Integration & Windows support Ritta Narita Fluentd Meetup 2016
  2. 2. About me. Ritta Narita joined Treasure Data after internship in 2015 2 @naritta @narittan
  3. 3. Fluentd v0.14 ServerEngine is integrated
  4. 4. What’s ServerEngine? A framework to implement robust multiprocess servers like Unicorn worker worker worker super visor Server Dynamic reconfig
 live restart Heartbeat via pipe auto restart multi process/thread
  5. 5. how to use ServerEngine worker super visor Server module MyServer def before_run @sock = TCPServer.new( config[:bind], config[:port]) end attr_reader :sock end module MyWorker def run c = server.sock.accept c.write "Awesome work!" end def stop c.close end end 1. write server module and worker module
  6. 6. how to use ServerEngine worker super visor Server se = ServerEngine.create(MyServer, MyWorker, { daemonize: true, log: 'myserver.log', pid_path: 'myserver.pid', worker_type: 'process', workers: 4, bind: '0.0.0.0', port: 9071, }) se.run 2. write configuration and run
  7. 7. module MyWorker def spawn(process_manager) process_manager.spawn(env, config[:spawn_cmd]) end end ServerEngine.create(MyServer, MyWorker, { worker_type:‘spawn’, spawn_cmd: cmd }) ServerEngine.create(MyServer, MyWorker, { worker_type:‘thread' }) workerServer workerServer workerServer spawn thread processServerEngine.create(MyServer, MyWorker, { worker_type:‘process' }) worker type
  8. 8. What’s the advantages for integrating serverengine into fluentd?
  9. 9. serverengine advantages Auto restart live restart socket manager signal handler log rotation
  10. 10. Auto restart live restart socket manager signal handler log rotation serverengine advantages
  11. 11. if worker died unexpectedly… worker worker super visor Server worker
  12. 12. if worker died unexpectedly… worker worker worker super visor Server
  13. 13. if worker died unexpectedly… worker worker super visor Server It’s detected by heartbeat and server launch worker again. worker
  14. 14. Auto restart live restart socket manager signal handler log rotation serverengine advantages
  15. 15. worker super visor Server tcp tcp send SIGHUP live restart
  16. 16. super visor Server worker tcp kill the worker, leaving server’s condition live restart
  17. 17. worker super visor Server worker tcp start new worker and read config again live restart
  18. 18. worker super visor Server worker tcp tcp share the TCP with living zero downtime live restart
  19. 19. Signal and RPC server api/config.reloadlive restart/config reload HUP /api/processes .interruptWorkers INT /api/processes .killWorkers /api/plugins. flushBuffers system config reload USR2 TERM USR1flush buffers shutdown
  20. 20. worker super visor Server you can reload system config without restart system config reload send USR2 check mtime and previous load time, if there is new one, reload! serverengine config cash new config
  21. 21. Auto restart live restart socket manager signal handler log rotation serverengine advantages
  22. 22. Socket Manager worker Socket Manager client Socket Manager server Server 26 ①socket listen request②find or create listening socket ③share the socket FD(unix)/ duplicated socket info(windows) UDS(unix)/ TCP(windows) ④get socket and accept in worker
  23. 23. how to use Socket Manager worker Socket Manager client Socket Manager server Server 27 #generate path for UDS/TCP for server connection socket_manager_path = ServerEngine::SocketManager::Server.generate_path #open socket manager server socket_manager_server = ServerEngine::SocketManager::Server.open(socket_manager_path) In server
  24. 24. worker Socket Manager client Socket Manager server Server 28 #get socket manager client from socket manager path socket_manager = ServerEngine::SocketManager::Client.new(socket_manager_path) #get listening socket from socket manager lsock = socket_manager.listen_tcp(bind, port) In worker how to use Socket Manager
  25. 25. What’s the merit of Socket Manager? worker worker worker Socket Manager client Socket Manager client Socket Manager client Socket Manager server super visor Server 29 we can use multicore easily in future
  26. 26. easy to use multicore power fully 30 with SocketManagerwith in_multiprocess plugin <source> type multiprocess <process> cmdline -c /etc/td-agent/td-agent-child1.conf </process> <process> cmdline -c /etc/td-agent/td-agent-child2.conf </process> </source> #/etc/td-agent/td-agent-child1.conf <source> type forward port 24224 </source> #/etc/td-agent/td-agent-child2.conf <source> type forward port 24225 </source> <source> type forward port 24224 </source> setting when using 2 core
  27. 27. Auto restart live restart socket manager signal handler log rotation serverengine advantages
  28. 28. Queue based signal handling When user sent USR1 and QUIT… signal handler thread user USR1 send USR1
  29. 29. Queue based signal handling When user sent USR1 and QUIT… signal handler thread user USR1 USR1
  30. 30. Queue based signal handling When user sent USR1 and QUIT… signal handler thread user QUIT USR1 send QUIT
  31. 31. Queue based signal handling When user sent USR1 and QUIT… signal handler thread user USR1 QUIT
  32. 32. Queue based signal handling When user sent USR1 and QUIT… signal handler thread user USR1 QUIT run!
  33. 33. Queue based signal handling When user sent USR1 and QUIT… signal handler thread user QUIT
  34. 34. Queue based signal handling When user sent USR1 and QUIT… signal handler thread user QUIT run!
  35. 35. Auto restart live restart socket manager signal handler log rotation serverengine advantages
  36. 36. support new features (These are already ported to ruby core.) ・multiprocess aware log rotation ・reopening of log file ・'trace' level, which is lower level than ‘debug' ServerEngine.create(MyServer, MyWorker, { log: 'myserver.log', log_level: 'debug', log_rotate_age: 5, log_rotate_size: 1*1024*1024, })
  37. 37. summary windows ・signal → rpc handler ・no socket FD and UDS ・fork based → spawn based ・live & auto restart ・multicore with socket manager ・sophisticated signal handler ・easier log rotation
  38. 38. Thank you!
  39. 39. appendix
  40. 40. unix→fork based damon windows→spawn based damon damonize fluentd server engine ①set windows_daemon_cmdline in configuration ②run windows_daemon_cmdline when damonize ③windows_daemon_cmdline runs daemon.rb ④run_server with servermodule and workermodule

×