SlideShare uma empresa Scribd logo
1 de 27
Baixar para ler offline
MySQL Proxy



    Making MySQL more flexible
          Jan Kneschke
         jan@mysql.com



                 
MySQL Proxy
    ●
        proxy­servers forward requests to backends 
        and can
        –   transform, handle or block them
    ●
        released under the GPL
        –   see http://forge.mysql.com/wiki/MySQL_Proxy
    ●
        developed as part of the Enterprise Tools since 
        February 2007
                                   
Design Decisions
    ●
        goal is to be transparent to the application layer
    ●
        supports all platforms and languages
    ●
        designed to handle thousands of parallel 
        connections (c10k)
    ●
        uses a embedded scripting language for 
        customizations


                                
Transparency
    ●
        SHOW WARNINGS can be worked around with 
        Query Injection
    ●
        SELECT USER() shows the connected user 
        (the proxy, not the client) which can be 
        corrected with result­set rewriting
    ●
        host auth against the MySQL server


                              
Latency
    ●
        early tests via localhost
    ●
        same script run directly and through the proxy
    ●
        latency per mysql­packet: 0.4ms
    ●
        ping RTT on 1Gbit: 0.1ms




                                 
Load Balancing
    ●
        load balancing distributes the load across 
        several slaves
    ●
        Shortest Queue First is default
        –   send new connections to the server with the least 
            number of open connections




                                    
Fail Over
    ●
        dead host are detected
    ●
        taking out of load balancing for 2min
    ●
        uses custom load balancers to decide how to 
        handle a dead host
        –   hot + standby
        –   uses load balancing


                                   
Removing SPoF
    ●
        one Proxy == Single Point of Failure
    ●
        use external Heartbeat (linuxha.org) or
    ●
        2 LB proxies + 1 Host Selecting Proxy per 
        application server




                               
Failsafe Load Balancing




                
Flexibility
    ●
        proxy embeds LUA 
    ●
        allows analyzing and manipulating packets 
        –   Inspection
        –   Rewriting
        –   Blocking
        –   Injection


                               
LUA
    ●
        PiL http://lua.org/manual/5.1/
    ●
        embedded, simple, efficient
    ●
        can do OO­like programming
    ●
        has scalars, tables, metatables and anonymous 
        functions



                                
Query Rewriting
    ●
        Macro Packages (ls, cd, who, ...)
    ●
        tagging queries with SQL_CACHE
    ●
        migrating table­names and SQL dialects
    ●
        turn EXPLAIN UPDATE|DELETE  into 
        equivalent EXPLAIN SELECT



                              
Query Profiling
    ●
        SHOW SESSION STATUS around a Query
    Exec_time: 6749 us
    .. Handler_read_rnd_next = 252
    .. Handler_write = 252
    .. Select_scan = 1




                          
Query Statistics
    ●
        Normalize Queries to track query usage
    ●
        Count Table and Index usage
    ●
        Optimize Query Cache Usage by injecting 
        SQL_CACHE in cachable queries
    ●
        see lib/analyze­queries.lua



                              
Auditing
    ●
        Diagnostic Auditing
    ●
        track which user+ip run which query or 
        accessed which objects when
    ●
        assign query­costs
    ●
        log gathered information in a central place
    ●
        see lib/auditing.lua

                                
Global Transaction ID
    ●
        Inject a counter in all transactions
    ●
        Answers questions like
        –   which slave is most current
        –   can I read from this slave, or do I have to read from 
            master
        –   you name it



                                     
Connection Pooling
    ●
        reusing open connections between proxy and 
        server
    ●
        reduces concurrency on the MySQL Server
    ●
        external connection pool for PHP




                               
Statement Routing
    ●
        split the query stream into reading and writing
        –   READs go to the slaves
        –   WRITEs and transactions to the master
    ●
        automatic scale­out
    ●
        sharding



                                      
Tokenizer
    ●
        turns a SQL query into a token stream
    ●
        not a full parser, just a tokenizer for speed 
        reasons
    ●
        understands KEYWORDS, /*comments*/, 
        “strings”, 123 and `literals`
    ●
        later we'll add support for SQL modes


                                 
normalizing Queries

    1:  { TK_SQL_SELECT, select }
    2:  { TK_STAR, * }
    3:  { TK_SQL_FROM, from }
    4:  { TK_LITERAL, t1 }
    5:  { TK_SQL_WHERE, where }
    6:  { TK_LITERAL, id }
    7:  { TK_EQ, = }
    8:  { TK_INTEGER, 1 }
    normalized query: SELECT * FROM `t1` WHERE 
    `id` = ? 



                            
Libraries
    ●
        auto­config                  ●
                                         parser
        –   SET GLOBAL ...               –   extract tablenames
    ●
        balance                      ●
                                         tokenizer
        –   load balancers               –   normalize()
    ●
        commands                         –   cleanup queries
        –   parse MySQL 
            Command Packets
                                  
Internals – LUA scripting
    ●
        proxy.* is the namespace
    ●
        proxy.connection.* is the current 
        connection
    ●
        proxy.backends[...] are the backends
    ●
        proxy.global.* is the global table
    ●
        proxy.global.config.* is used for the 
        config
                             
Internals ­ Scope
    ●
        Each connection has its own script scope
    ●
        proxy.global.* to share data between 
        connections
    ●
        use local to make variables local to the 
        function
    ●
        use package.seeall() to export functions 
        from modules
                               
Internals ­ Threading
    ●
        the global scope and threading don't play nice 
        by default
    ●
        http://www.cs.princeton.edu/~diego/professional/lua
    ●
        patches lua to apply mutexes around variable 
        access




                               
Internals – Script Cache
    ●
        0.6.0 we reload the script on each connection 
        start
    ●
        adding a script cache with mtime check
    ●
        lua_pushvalue(L, ­1) does the trick




                               
Roadmap
    ●
        to be released 0.6.0
        –   tokenizer
        –   read­write splitting
        –   Query Statistics
    ●
        later
        –   parallel Queries
        –   proxy initiates connections
                                    
LUA ­ Gotchas
    ●
        only false and nil are !true, 0 is true
    ●
        to say “not equal” you use ~=
    ●
        there are no shortcuts
        –   no a++, no a *= 4, ...
        –   no a > b ? a : b (there is “(a > b) and a or b)




                                      

Mais conteúdo relacionado

Mais procurados

Using OVSDB and OpenFlow southbound plugins
Using OVSDB and OpenFlow southbound pluginsUsing OVSDB and OpenFlow southbound plugins
Using OVSDB and OpenFlow southbound pluginsOpenDaylight
 
Open vswitch datapath implementation
Open vswitch datapath implementationOpen vswitch datapath implementation
Open vswitch datapath implementationVishal Kapoor
 
Apache zookeeper seminar_trinh_viet_dung_03_2016
Apache zookeeper seminar_trinh_viet_dung_03_2016Apache zookeeper seminar_trinh_viet_dung_03_2016
Apache zookeeper seminar_trinh_viet_dung_03_2016Viet-Dung TRINH
 
Streaming replication in practice
Streaming replication in practiceStreaming replication in practice
Streaming replication in practiceAlexey Lesovsky
 
Logging with Logback in Scala
Logging with Logback in ScalaLogging with Logback in Scala
Logging with Logback in ScalaKnoldus Inc.
 
High performance messaging with Apache Pulsar
High performance messaging with Apache PulsarHigh performance messaging with Apache Pulsar
High performance messaging with Apache PulsarMatteo Merli
 
Modern Distributed Messaging and RPC
Modern Distributed Messaging and RPCModern Distributed Messaging and RPC
Modern Distributed Messaging and RPCMax Alexejev
 
Linux Cluster next generation
Linux Cluster next generationLinux Cluster next generation
Linux Cluster next generationsamsolutionsby
 
Logging with Logback in Scala
Logging with Logback in ScalaLogging with Logback in Scala
Logging with Logback in ScalaKnoldus Inc.
 
Using Apache Pulsar as a Modern, Scalable, High Performing JMS Platform - Pus...
Using Apache Pulsar as a Modern, Scalable, High Performing JMS Platform - Pus...Using Apache Pulsar as a Modern, Scalable, High Performing JMS Platform - Pus...
Using Apache Pulsar as a Modern, Scalable, High Performing JMS Platform - Pus...StreamNative
 
Streaming Processing with a Distributed Commit Log
Streaming Processing with a Distributed Commit LogStreaming Processing with a Distributed Commit Log
Streaming Processing with a Distributed Commit LogJoe Stein
 
Mystery Machine Overview
Mystery Machine OverviewMystery Machine Overview
Mystery Machine OverviewIvan Glushkov
 
A user's perspective on SaltStack and other configuration management tools
A user's perspective on SaltStack and other configuration management toolsA user's perspective on SaltStack and other configuration management tools
A user's perspective on SaltStack and other configuration management toolsSaltStack
 
pgpool: Features and Development
pgpool: Features and Developmentpgpool: Features and Development
pgpool: Features and Developmentelliando dias
 
Terraform 0.13: Rise of the modules
Terraform 0.13: Rise of the modulesTerraform 0.13: Rise of the modules
Terraform 0.13: Rise of the modulesMarko Bevc
 
Nov. 4, 2011 o reilly webcast-hbase- lars george
Nov. 4, 2011 o reilly webcast-hbase- lars georgeNov. 4, 2011 o reilly webcast-hbase- lars george
Nov. 4, 2011 o reilly webcast-hbase- lars georgeO'Reilly Media
 
Introduction to Apache ZooKeeper | Big Data Hadoop Spark Tutorial | CloudxLab
Introduction to Apache ZooKeeper | Big Data Hadoop Spark Tutorial | CloudxLabIntroduction to Apache ZooKeeper | Big Data Hadoop Spark Tutorial | CloudxLab
Introduction to Apache ZooKeeper | Big Data Hadoop Spark Tutorial | CloudxLabCloudxLab
 
What you need to know for postgresql operation
What you need to know for postgresql operationWhat you need to know for postgresql operation
What you need to know for postgresql operationAnton Bushmelev
 

Mais procurados (20)

Using OVSDB and OpenFlow southbound plugins
Using OVSDB and OpenFlow southbound pluginsUsing OVSDB and OpenFlow southbound plugins
Using OVSDB and OpenFlow southbound plugins
 
Open vswitch datapath implementation
Open vswitch datapath implementationOpen vswitch datapath implementation
Open vswitch datapath implementation
 
Apache zookeeper seminar_trinh_viet_dung_03_2016
Apache zookeeper seminar_trinh_viet_dung_03_2016Apache zookeeper seminar_trinh_viet_dung_03_2016
Apache zookeeper seminar_trinh_viet_dung_03_2016
 
Streaming replication in practice
Streaming replication in practiceStreaming replication in practice
Streaming replication in practice
 
Logging with Logback in Scala
Logging with Logback in ScalaLogging with Logback in Scala
Logging with Logback in Scala
 
High performance messaging with Apache Pulsar
High performance messaging with Apache PulsarHigh performance messaging with Apache Pulsar
High performance messaging with Apache Pulsar
 
Jsp project module
Jsp project moduleJsp project module
Jsp project module
 
Modern Distributed Messaging and RPC
Modern Distributed Messaging and RPCModern Distributed Messaging and RPC
Modern Distributed Messaging and RPC
 
Linux Cluster next generation
Linux Cluster next generationLinux Cluster next generation
Linux Cluster next generation
 
Logging with Logback in Scala
Logging with Logback in ScalaLogging with Logback in Scala
Logging with Logback in Scala
 
Using Apache Pulsar as a Modern, Scalable, High Performing JMS Platform - Pus...
Using Apache Pulsar as a Modern, Scalable, High Performing JMS Platform - Pus...Using Apache Pulsar as a Modern, Scalable, High Performing JMS Platform - Pus...
Using Apache Pulsar as a Modern, Scalable, High Performing JMS Platform - Pus...
 
Streaming Processing with a Distributed Commit Log
Streaming Processing with a Distributed Commit LogStreaming Processing with a Distributed Commit Log
Streaming Processing with a Distributed Commit Log
 
Mystery Machine Overview
Mystery Machine OverviewMystery Machine Overview
Mystery Machine Overview
 
Vikentsi Lapa - Tools for testing
Vikentsi Lapa - Tools for testingVikentsi Lapa - Tools for testing
Vikentsi Lapa - Tools for testing
 
A user's perspective on SaltStack and other configuration management tools
A user's perspective on SaltStack and other configuration management toolsA user's perspective on SaltStack and other configuration management tools
A user's perspective on SaltStack and other configuration management tools
 
pgpool: Features and Development
pgpool: Features and Developmentpgpool: Features and Development
pgpool: Features and Development
 
Terraform 0.13: Rise of the modules
Terraform 0.13: Rise of the modulesTerraform 0.13: Rise of the modules
Terraform 0.13: Rise of the modules
 
Nov. 4, 2011 o reilly webcast-hbase- lars george
Nov. 4, 2011 o reilly webcast-hbase- lars georgeNov. 4, 2011 o reilly webcast-hbase- lars george
Nov. 4, 2011 o reilly webcast-hbase- lars george
 
Introduction to Apache ZooKeeper | Big Data Hadoop Spark Tutorial | CloudxLab
Introduction to Apache ZooKeeper | Big Data Hadoop Spark Tutorial | CloudxLabIntroduction to Apache ZooKeeper | Big Data Hadoop Spark Tutorial | CloudxLab
Introduction to Apache ZooKeeper | Big Data Hadoop Spark Tutorial | CloudxLab
 
What you need to know for postgresql operation
What you need to know for postgresql operationWhat you need to know for postgresql operation
What you need to know for postgresql operation
 

Destaque

Erlang全接触
Erlang全接触Erlang全接触
Erlang全接触Feng Yu
 
Erlang Optimize
Erlang OptimizeErlang Optimize
Erlang OptimizeFeng Yu
 
高级服务器设计和实现1
高级服务器设计和实现1高级服务器设计和实现1
高级服务器设计和实现1Feng Yu
 
Inside Erlang Vm II
Inside Erlang Vm IIInside Erlang Vm II
Inside Erlang Vm IIFeng Yu
 
Erlang Emulator Implementation
Erlang Emulator ImplementationErlang Emulator Implementation
Erlang Emulator ImplementationFeng Yu
 
Oprofile linux
Oprofile linuxOprofile linux
Oprofile linuxFeng Yu
 
Erlang low cost_clound_computing
Erlang low cost_clound_computingErlang low cost_clound_computing
Erlang low cost_clound_computingFeng Yu
 
高性能集群服务器(Erlang解决方案)
高性能集群服务器(Erlang解决方案)高性能集群服务器(Erlang解决方案)
高性能集群服务器(Erlang解决方案)Feng Yu
 
了解网络
了解网络了解网络
了解网络Feng Yu
 
mnesia脑裂问题综述
mnesia脑裂问题综述mnesia脑裂问题综述
mnesia脑裂问题综述Feng Yu
 
淘宝商品库MySQL优化实践
淘宝商品库MySQL优化实践淘宝商品库MySQL优化实践
淘宝商品库MySQL优化实践Feng Yu
 
Systemtap
SystemtapSystemtap
SystemtapFeng Yu
 
Erlang分布式系统的的领域语言
Erlang分布式系统的的领域语言Erlang分布式系统的的领域语言
Erlang分布式系统的的领域语言Feng Yu
 
C1000K高性能服务器构建技术
C1000K高性能服务器构建技术C1000K高性能服务器构建技术
C1000K高性能服务器构建技术Feng Yu
 

Destaque (14)

Erlang全接触
Erlang全接触Erlang全接触
Erlang全接触
 
Erlang Optimize
Erlang OptimizeErlang Optimize
Erlang Optimize
 
高级服务器设计和实现1
高级服务器设计和实现1高级服务器设计和实现1
高级服务器设计和实现1
 
Inside Erlang Vm II
Inside Erlang Vm IIInside Erlang Vm II
Inside Erlang Vm II
 
Erlang Emulator Implementation
Erlang Emulator ImplementationErlang Emulator Implementation
Erlang Emulator Implementation
 
Oprofile linux
Oprofile linuxOprofile linux
Oprofile linux
 
Erlang low cost_clound_computing
Erlang low cost_clound_computingErlang low cost_clound_computing
Erlang low cost_clound_computing
 
高性能集群服务器(Erlang解决方案)
高性能集群服务器(Erlang解决方案)高性能集群服务器(Erlang解决方案)
高性能集群服务器(Erlang解决方案)
 
了解网络
了解网络了解网络
了解网络
 
mnesia脑裂问题综述
mnesia脑裂问题综述mnesia脑裂问题综述
mnesia脑裂问题综述
 
淘宝商品库MySQL优化实践
淘宝商品库MySQL优化实践淘宝商品库MySQL优化实践
淘宝商品库MySQL优化实践
 
Systemtap
SystemtapSystemtap
Systemtap
 
Erlang分布式系统的的领域语言
Erlang分布式系统的的领域语言Erlang分布式系统的的领域语言
Erlang分布式系统的的领域语言
 
C1000K高性能服务器构建技术
C1000K高性能服务器构建技术C1000K高性能服务器构建技术
C1000K高性能服务器构建技术
 

Semelhante a My Sql Proxy

Buytaert kris my_sql-pacemaker
Buytaert kris my_sql-pacemakerBuytaert kris my_sql-pacemaker
Buytaert kris my_sql-pacemakerkuchinskaya
 
Scaling Up Logging and Metrics
Scaling Up Logging and MetricsScaling Up Logging and Metrics
Scaling Up Logging and MetricsRicardo Lourenço
 
Mysqlhacodebits20091203 1260184765-phpapp02
Mysqlhacodebits20091203 1260184765-phpapp02Mysqlhacodebits20091203 1260184765-phpapp02
Mysqlhacodebits20091203 1260184765-phpapp02Louis liu
 
MySQL High Availability Solutions
MySQL High Availability SolutionsMySQL High Availability Solutions
MySQL High Availability SolutionsLenz Grimmer
 
MySQL High Availability Solutions
MySQL High Availability SolutionsMySQL High Availability Solutions
MySQL High Availability SolutionsLenz Grimmer
 
Eko10 Workshop Opensource Database Auditing
Eko10  Workshop Opensource Database AuditingEko10  Workshop Opensource Database Auditing
Eko10 Workshop Opensource Database AuditingJuan Berner
 
Comparison between zookeeper, etcd 3 and other distributed coordination systems
Comparison between zookeeper, etcd 3 and other distributed coordination systemsComparison between zookeeper, etcd 3 and other distributed coordination systems
Comparison between zookeeper, etcd 3 and other distributed coordination systemsImesha Sudasingha
 
MySQL X protocol - Talking to MySQL Directly over the Wire
MySQL X protocol - Talking to MySQL Directly over the WireMySQL X protocol - Talking to MySQL Directly over the Wire
MySQL X protocol - Talking to MySQL Directly over the WireSimon J Mudd
 
Eko10 workshop - OPEN SOURCE DATABASE MONITORING
Eko10 workshop - OPEN SOURCE DATABASE MONITORINGEko10 workshop - OPEN SOURCE DATABASE MONITORING
Eko10 workshop - OPEN SOURCE DATABASE MONITORINGPablo Garbossa
 
NetflixOSS Open House Lightning talks
NetflixOSS Open House Lightning talksNetflixOSS Open House Lightning talks
NetflixOSS Open House Lightning talksRuslan Meshenberg
 
Multi-Tenancy Kafka cluster for LINE services with 250 billion daily messages
Multi-Tenancy Kafka cluster for LINE services with 250 billion daily messagesMulti-Tenancy Kafka cluster for LINE services with 250 billion daily messages
Multi-Tenancy Kafka cluster for LINE services with 250 billion daily messagesLINE Corporation
 
Microsoft Hekaton
Microsoft HekatonMicrosoft Hekaton
Microsoft HekatonSiraj Memon
 
Tips & Tricks for Apache Kafka®
Tips & Tricks for Apache Kafka®Tips & Tricks for Apache Kafka®
Tips & Tricks for Apache Kafka®confluent
 
Strimzi - Where Apache Kafka meets OpenShift - OpenShift Spain MeetUp
Strimzi - Where Apache Kafka meets OpenShift - OpenShift Spain MeetUpStrimzi - Where Apache Kafka meets OpenShift - OpenShift Spain MeetUp
Strimzi - Where Apache Kafka meets OpenShift - OpenShift Spain MeetUpJosé Román Martín Gil
 
Making MySQL Administration a Breeze - A look into a MySQL DBA's toolchest
Making MySQL Administration a Breeze - A look into a MySQL DBA's toolchest Making MySQL Administration a Breeze - A look into a MySQL DBA's toolchest
Making MySQL Administration a Breeze - A look into a MySQL DBA's toolchest Lenz Grimmer
 
Open stack HA - Theory to Reality
Open stack HA -  Theory to RealityOpen stack HA -  Theory to Reality
Open stack HA - Theory to RealitySriram Subramanian
 
14th Athens Big Data Meetup - Landoop Workshop - Apache Kafka Entering The St...
14th Athens Big Data Meetup - Landoop Workshop - Apache Kafka Entering The St...14th Athens Big Data Meetup - Landoop Workshop - Apache Kafka Entering The St...
14th Athens Big Data Meetup - Landoop Workshop - Apache Kafka Entering The St...Athens Big Data
 
Apache spark - Installation
Apache spark - InstallationApache spark - Installation
Apache spark - InstallationMartin Zapletal
 
Deploying containers and managing them on multiple Docker hosts, Docker Meetu...
Deploying containers and managing them on multiple Docker hosts, Docker Meetu...Deploying containers and managing them on multiple Docker hosts, Docker Meetu...
Deploying containers and managing them on multiple Docker hosts, Docker Meetu...dotCloud
 

Semelhante a My Sql Proxy (20)

Buytaert kris my_sql-pacemaker
Buytaert kris my_sql-pacemakerBuytaert kris my_sql-pacemaker
Buytaert kris my_sql-pacemaker
 
Scaling Up Logging and Metrics
Scaling Up Logging and MetricsScaling Up Logging and Metrics
Scaling Up Logging and Metrics
 
The Accidental DBA
The Accidental DBAThe Accidental DBA
The Accidental DBA
 
Mysqlhacodebits20091203 1260184765-phpapp02
Mysqlhacodebits20091203 1260184765-phpapp02Mysqlhacodebits20091203 1260184765-phpapp02
Mysqlhacodebits20091203 1260184765-phpapp02
 
MySQL High Availability Solutions
MySQL High Availability SolutionsMySQL High Availability Solutions
MySQL High Availability Solutions
 
MySQL High Availability Solutions
MySQL High Availability SolutionsMySQL High Availability Solutions
MySQL High Availability Solutions
 
Eko10 Workshop Opensource Database Auditing
Eko10  Workshop Opensource Database AuditingEko10  Workshop Opensource Database Auditing
Eko10 Workshop Opensource Database Auditing
 
Comparison between zookeeper, etcd 3 and other distributed coordination systems
Comparison between zookeeper, etcd 3 and other distributed coordination systemsComparison between zookeeper, etcd 3 and other distributed coordination systems
Comparison between zookeeper, etcd 3 and other distributed coordination systems
 
MySQL X protocol - Talking to MySQL Directly over the Wire
MySQL X protocol - Talking to MySQL Directly over the WireMySQL X protocol - Talking to MySQL Directly over the Wire
MySQL X protocol - Talking to MySQL Directly over the Wire
 
Eko10 workshop - OPEN SOURCE DATABASE MONITORING
Eko10 workshop - OPEN SOURCE DATABASE MONITORINGEko10 workshop - OPEN SOURCE DATABASE MONITORING
Eko10 workshop - OPEN SOURCE DATABASE MONITORING
 
NetflixOSS Open House Lightning talks
NetflixOSS Open House Lightning talksNetflixOSS Open House Lightning talks
NetflixOSS Open House Lightning talks
 
Multi-Tenancy Kafka cluster for LINE services with 250 billion daily messages
Multi-Tenancy Kafka cluster for LINE services with 250 billion daily messagesMulti-Tenancy Kafka cluster for LINE services with 250 billion daily messages
Multi-Tenancy Kafka cluster for LINE services with 250 billion daily messages
 
Microsoft Hekaton
Microsoft HekatonMicrosoft Hekaton
Microsoft Hekaton
 
Tips & Tricks for Apache Kafka®
Tips & Tricks for Apache Kafka®Tips & Tricks for Apache Kafka®
Tips & Tricks for Apache Kafka®
 
Strimzi - Where Apache Kafka meets OpenShift - OpenShift Spain MeetUp
Strimzi - Where Apache Kafka meets OpenShift - OpenShift Spain MeetUpStrimzi - Where Apache Kafka meets OpenShift - OpenShift Spain MeetUp
Strimzi - Where Apache Kafka meets OpenShift - OpenShift Spain MeetUp
 
Making MySQL Administration a Breeze - A look into a MySQL DBA's toolchest
Making MySQL Administration a Breeze - A look into a MySQL DBA's toolchest Making MySQL Administration a Breeze - A look into a MySQL DBA's toolchest
Making MySQL Administration a Breeze - A look into a MySQL DBA's toolchest
 
Open stack HA - Theory to Reality
Open stack HA -  Theory to RealityOpen stack HA -  Theory to Reality
Open stack HA - Theory to Reality
 
14th Athens Big Data Meetup - Landoop Workshop - Apache Kafka Entering The St...
14th Athens Big Data Meetup - Landoop Workshop - Apache Kafka Entering The St...14th Athens Big Data Meetup - Landoop Workshop - Apache Kafka Entering The St...
14th Athens Big Data Meetup - Landoop Workshop - Apache Kafka Entering The St...
 
Apache spark - Installation
Apache spark - InstallationApache spark - Installation
Apache spark - Installation
 
Deploying containers and managing them on multiple Docker hosts, Docker Meetu...
Deploying containers and managing them on multiple Docker hosts, Docker Meetu...Deploying containers and managing them on multiple Docker hosts, Docker Meetu...
Deploying containers and managing them on multiple Docker hosts, Docker Meetu...
 

Último

DUST OF SNOW_BY ROBERT FROST_EDITED BY_ TANMOY MISHRA
DUST OF SNOW_BY ROBERT FROST_EDITED BY_ TANMOY MISHRADUST OF SNOW_BY ROBERT FROST_EDITED BY_ TANMOY MISHRA
DUST OF SNOW_BY ROBERT FROST_EDITED BY_ TANMOY MISHRATanmoy Mishra
 
HED Office Sohayok Exam Question Solution 2023.pdf
HED Office Sohayok Exam Question Solution 2023.pdfHED Office Sohayok Exam Question Solution 2023.pdf
HED Office Sohayok Exam Question Solution 2023.pdfMohonDas
 
The Singapore Teaching Practice document
The Singapore Teaching Practice documentThe Singapore Teaching Practice document
The Singapore Teaching Practice documentXsasf Sfdfasd
 
CapTechU Doctoral Presentation -March 2024 slides.pptx
CapTechU Doctoral Presentation -March 2024 slides.pptxCapTechU Doctoral Presentation -March 2024 slides.pptx
CapTechU Doctoral Presentation -March 2024 slides.pptxCapitolTechU
 
3.21.24 The Origins of Black Power.pptx
3.21.24  The Origins of Black Power.pptx3.21.24  The Origins of Black Power.pptx
3.21.24 The Origins of Black Power.pptxmary850239
 
The basics of sentences session 10pptx.pptx
The basics of sentences session 10pptx.pptxThe basics of sentences session 10pptx.pptx
The basics of sentences session 10pptx.pptxheathfieldcps1
 
How to Show Error_Warning Messages in Odoo 17
How to Show Error_Warning Messages in Odoo 17How to Show Error_Warning Messages in Odoo 17
How to Show Error_Warning Messages in Odoo 17Celine George
 
Quality Assurance_GOOD LABORATORY PRACTICE
Quality Assurance_GOOD LABORATORY PRACTICEQuality Assurance_GOOD LABORATORY PRACTICE
Quality Assurance_GOOD LABORATORY PRACTICESayali Powar
 
Patient Counselling. Definition of patient counseling; steps involved in pati...
Patient Counselling. Definition of patient counseling; steps involved in pati...Patient Counselling. Definition of patient counseling; steps involved in pati...
Patient Counselling. Definition of patient counseling; steps involved in pati...raviapr7
 
General views of Histopathology and step
General views of Histopathology and stepGeneral views of Histopathology and step
General views of Histopathology and stepobaje godwin sunday
 
Ultra structure and life cycle of Plasmodium.pptx
Ultra structure and life cycle of Plasmodium.pptxUltra structure and life cycle of Plasmodium.pptx
Ultra structure and life cycle of Plasmodium.pptxDr. Asif Anas
 
CAULIFLOWER BREEDING 1 Parmar pptx
CAULIFLOWER BREEDING 1 Parmar pptxCAULIFLOWER BREEDING 1 Parmar pptx
CAULIFLOWER BREEDING 1 Parmar pptxSaurabhParmar42
 
Practical Research 1: Lesson 8 Writing the Thesis Statement.pptx
Practical Research 1: Lesson 8 Writing the Thesis Statement.pptxPractical Research 1: Lesson 8 Writing the Thesis Statement.pptx
Practical Research 1: Lesson 8 Writing the Thesis Statement.pptxKatherine Villaluna
 
Diploma in Nursing Admission Test Question Solution 2023.pdf
Diploma in Nursing Admission Test Question Solution 2023.pdfDiploma in Nursing Admission Test Question Solution 2023.pdf
Diploma in Nursing Admission Test Question Solution 2023.pdfMohonDas
 
How to Solve Singleton Error in the Odoo 17
How to Solve Singleton Error in the  Odoo 17How to Solve Singleton Error in the  Odoo 17
How to Solve Singleton Error in the Odoo 17Celine George
 
Prescribed medication order and communication skills.pptx
Prescribed medication order and communication skills.pptxPrescribed medication order and communication skills.pptx
Prescribed medication order and communication skills.pptxraviapr7
 
Maximizing Impact_ Nonprofit Website Planning, Budgeting, and Design.pdf
Maximizing Impact_ Nonprofit Website Planning, Budgeting, and Design.pdfMaximizing Impact_ Nonprofit Website Planning, Budgeting, and Design.pdf
Maximizing Impact_ Nonprofit Website Planning, Budgeting, and Design.pdfTechSoup
 
Practical Research 1 Lesson 9 Scope and delimitation.pptx
Practical Research 1 Lesson 9 Scope and delimitation.pptxPractical Research 1 Lesson 9 Scope and delimitation.pptx
Practical Research 1 Lesson 9 Scope and delimitation.pptxKatherine Villaluna
 
Patterns of Written Texts Across Disciplines.pptx
Patterns of Written Texts Across Disciplines.pptxPatterns of Written Texts Across Disciplines.pptx
Patterns of Written Texts Across Disciplines.pptxMYDA ANGELICA SUAN
 
How to Use api.constrains ( ) in Odoo 17
How to Use api.constrains ( ) in Odoo 17How to Use api.constrains ( ) in Odoo 17
How to Use api.constrains ( ) in Odoo 17Celine George
 

Último (20)

DUST OF SNOW_BY ROBERT FROST_EDITED BY_ TANMOY MISHRA
DUST OF SNOW_BY ROBERT FROST_EDITED BY_ TANMOY MISHRADUST OF SNOW_BY ROBERT FROST_EDITED BY_ TANMOY MISHRA
DUST OF SNOW_BY ROBERT FROST_EDITED BY_ TANMOY MISHRA
 
HED Office Sohayok Exam Question Solution 2023.pdf
HED Office Sohayok Exam Question Solution 2023.pdfHED Office Sohayok Exam Question Solution 2023.pdf
HED Office Sohayok Exam Question Solution 2023.pdf
 
The Singapore Teaching Practice document
The Singapore Teaching Practice documentThe Singapore Teaching Practice document
The Singapore Teaching Practice document
 
CapTechU Doctoral Presentation -March 2024 slides.pptx
CapTechU Doctoral Presentation -March 2024 slides.pptxCapTechU Doctoral Presentation -March 2024 slides.pptx
CapTechU Doctoral Presentation -March 2024 slides.pptx
 
3.21.24 The Origins of Black Power.pptx
3.21.24  The Origins of Black Power.pptx3.21.24  The Origins of Black Power.pptx
3.21.24 The Origins of Black Power.pptx
 
The basics of sentences session 10pptx.pptx
The basics of sentences session 10pptx.pptxThe basics of sentences session 10pptx.pptx
The basics of sentences session 10pptx.pptx
 
How to Show Error_Warning Messages in Odoo 17
How to Show Error_Warning Messages in Odoo 17How to Show Error_Warning Messages in Odoo 17
How to Show Error_Warning Messages in Odoo 17
 
Quality Assurance_GOOD LABORATORY PRACTICE
Quality Assurance_GOOD LABORATORY PRACTICEQuality Assurance_GOOD LABORATORY PRACTICE
Quality Assurance_GOOD LABORATORY PRACTICE
 
Patient Counselling. Definition of patient counseling; steps involved in pati...
Patient Counselling. Definition of patient counseling; steps involved in pati...Patient Counselling. Definition of patient counseling; steps involved in pati...
Patient Counselling. Definition of patient counseling; steps involved in pati...
 
General views of Histopathology and step
General views of Histopathology and stepGeneral views of Histopathology and step
General views of Histopathology and step
 
Ultra structure and life cycle of Plasmodium.pptx
Ultra structure and life cycle of Plasmodium.pptxUltra structure and life cycle of Plasmodium.pptx
Ultra structure and life cycle of Plasmodium.pptx
 
CAULIFLOWER BREEDING 1 Parmar pptx
CAULIFLOWER BREEDING 1 Parmar pptxCAULIFLOWER BREEDING 1 Parmar pptx
CAULIFLOWER BREEDING 1 Parmar pptx
 
Practical Research 1: Lesson 8 Writing the Thesis Statement.pptx
Practical Research 1: Lesson 8 Writing the Thesis Statement.pptxPractical Research 1: Lesson 8 Writing the Thesis Statement.pptx
Practical Research 1: Lesson 8 Writing the Thesis Statement.pptx
 
Diploma in Nursing Admission Test Question Solution 2023.pdf
Diploma in Nursing Admission Test Question Solution 2023.pdfDiploma in Nursing Admission Test Question Solution 2023.pdf
Diploma in Nursing Admission Test Question Solution 2023.pdf
 
How to Solve Singleton Error in the Odoo 17
How to Solve Singleton Error in the  Odoo 17How to Solve Singleton Error in the  Odoo 17
How to Solve Singleton Error in the Odoo 17
 
Prescribed medication order and communication skills.pptx
Prescribed medication order and communication skills.pptxPrescribed medication order and communication skills.pptx
Prescribed medication order and communication skills.pptx
 
Maximizing Impact_ Nonprofit Website Planning, Budgeting, and Design.pdf
Maximizing Impact_ Nonprofit Website Planning, Budgeting, and Design.pdfMaximizing Impact_ Nonprofit Website Planning, Budgeting, and Design.pdf
Maximizing Impact_ Nonprofit Website Planning, Budgeting, and Design.pdf
 
Practical Research 1 Lesson 9 Scope and delimitation.pptx
Practical Research 1 Lesson 9 Scope and delimitation.pptxPractical Research 1 Lesson 9 Scope and delimitation.pptx
Practical Research 1 Lesson 9 Scope and delimitation.pptx
 
Patterns of Written Texts Across Disciplines.pptx
Patterns of Written Texts Across Disciplines.pptxPatterns of Written Texts Across Disciplines.pptx
Patterns of Written Texts Across Disciplines.pptx
 
How to Use api.constrains ( ) in Odoo 17
How to Use api.constrains ( ) in Odoo 17How to Use api.constrains ( ) in Odoo 17
How to Use api.constrains ( ) in Odoo 17
 

My Sql Proxy

  • 1. MySQL Proxy Making MySQL more flexible Jan Kneschke jan@mysql.com    
  • 2. MySQL Proxy ● proxy­servers forward requests to backends  and can – transform, handle or block them ● released under the GPL – see http://forge.mysql.com/wiki/MySQL_Proxy ● developed as part of the Enterprise Tools since  February 2007    
  • 3. Design Decisions ● goal is to be transparent to the application layer ● supports all platforms and languages ● designed to handle thousands of parallel  connections (c10k) ● uses a embedded scripting language for  customizations    
  • 4. Transparency ● SHOW WARNINGS can be worked around with  Query Injection ● SELECT USER() shows the connected user  (the proxy, not the client) which can be  corrected with result­set rewriting ● host auth against the MySQL server    
  • 5. Latency ● early tests via localhost ● same script run directly and through the proxy ● latency per mysql­packet: 0.4ms ● ping RTT on 1Gbit: 0.1ms    
  • 6. Load Balancing ● load balancing distributes the load across  several slaves ● Shortest Queue First is default – send new connections to the server with the least  number of open connections    
  • 7. Fail Over ● dead host are detected ● taking out of load balancing for 2min ● uses custom load balancers to decide how to  handle a dead host – hot + standby – uses load balancing    
  • 8. Removing SPoF ● one Proxy == Single Point of Failure ● use external Heartbeat (linuxha.org) or ● 2 LB proxies + 1 Host Selecting Proxy per  application server    
  • 10. Flexibility ● proxy embeds LUA  ● allows analyzing and manipulating packets  – Inspection – Rewriting – Blocking – Injection    
  • 11. LUA ● PiL http://lua.org/manual/5.1/ ● embedded, simple, efficient ● can do OO­like programming ● has scalars, tables, metatables and anonymous  functions    
  • 12. Query Rewriting ● Macro Packages (ls, cd, who, ...) ● tagging queries with SQL_CACHE ● migrating table­names and SQL dialects ● turn EXPLAIN UPDATE|DELETE  into  equivalent EXPLAIN SELECT    
  • 13. Query Profiling ● SHOW SESSION STATUS around a Query Exec_time: 6749 us .. Handler_read_rnd_next = 252 .. Handler_write = 252 .. Select_scan = 1    
  • 14. Query Statistics ● Normalize Queries to track query usage ● Count Table and Index usage ● Optimize Query Cache Usage by injecting  SQL_CACHE in cachable queries ● see lib/analyze­queries.lua    
  • 15. Auditing ● Diagnostic Auditing ● track which user+ip run which query or  accessed which objects when ● assign query­costs ● log gathered information in a central place ● see lib/auditing.lua    
  • 16. Global Transaction ID ● Inject a counter in all transactions ● Answers questions like – which slave is most current – can I read from this slave, or do I have to read from  master – you name it    
  • 17. Connection Pooling ● reusing open connections between proxy and  server ● reduces concurrency on the MySQL Server ● external connection pool for PHP    
  • 18. Statement Routing ● split the query stream into reading and writing – READs go to the slaves – WRITEs and transactions to the master ● automatic scale­out ● sharding    
  • 19. Tokenizer ● turns a SQL query into a token stream ● not a full parser, just a tokenizer for speed  reasons ● understands KEYWORDS, /*comments*/,  “strings”, 123 and `literals` ● later we'll add support for SQL modes    
  • 20. normalizing Queries 1:  { TK_SQL_SELECT, select } 2:  { TK_STAR, * } 3:  { TK_SQL_FROM, from } 4:  { TK_LITERAL, t1 } 5:  { TK_SQL_WHERE, where } 6:  { TK_LITERAL, id } 7:  { TK_EQ, = } 8:  { TK_INTEGER, 1 } normalized query: SELECT * FROM `t1` WHERE  `id` = ?     
  • 21. Libraries ● auto­config ● parser – SET GLOBAL ...  – extract tablenames ● balance ● tokenizer – load balancers – normalize() ● commands – cleanup queries – parse MySQL  Command Packets    
  • 22. Internals – LUA scripting ● proxy.* is the namespace ● proxy.connection.* is the current  connection ● proxy.backends[...] are the backends ● proxy.global.* is the global table ● proxy.global.config.* is used for the  config    
  • 23. Internals ­ Scope ● Each connection has its own script scope ● proxy.global.* to share data between  connections ● use local to make variables local to the  function ● use package.seeall() to export functions  from modules    
  • 24. Internals ­ Threading ● the global scope and threading don't play nice  by default ● http://www.cs.princeton.edu/~diego/professional/lua ● patches lua to apply mutexes around variable  access    
  • 25. Internals – Script Cache ● 0.6.0 we reload the script on each connection  start ● adding a script cache with mtime check ● lua_pushvalue(L, ­1) does the trick    
  • 26. Roadmap ● to be released 0.6.0 – tokenizer – read­write splitting – Query Statistics ● later – parallel Queries – proxy initiates connections    
  • 27. LUA ­ Gotchas ● only false and nil are !true, 0 is true ● to say “not equal” you use ~= ● there are no shortcuts – no a++, no a *= 4, ... – no a > b ? a : b (there is “(a > b) and a or b)