What to expect from MariaDB Platform X5, part 2

What to expect from
MariaDB Platform X5
(Part 2)
Todd Stoffel
Product Manager

What is MaxScale?
● Automatic failover
● Transaction replay
● Schema sharding
● Causal reads
● Query blocking
○ Database firewall
● Data masking
● Denial-of-service protection
○ Result limiting
Database proxy
●Offload replication from the primary
database
●Supports a very large number of read
replicas
Replication server

MariaDB MaxScale: New Features in 2.5
● New Graphic User Interface
(Alternative to MaxCtrl)
● New and Improved Binlog Router
● Stream Binlogs (as JSON) to Kafka
Broker
●Cooperative Monitoring and Locking
(New HA Solution)
●Distributed Cache between MaxScale
Servers (i.e. Redis, Memcached)
●Columnstore Orchestration

New Binlog Router
Allows MaxScale to serve replication traffic to any number of slaves
while removing that load from the true master
● Completely re-written
● More efficient and performant
● Heartbeat and burst interval configured automatically

Stream Binlogs (as JSON) to Kafka Broker
MariaDB
MaxScale
JSON Conversion
Kafka
App
ColumnStore

MaxScale 1
Active
Legend
Normal Traffic
MariaDB
MaxScale 2
Active
Redundant Traffic
App
+ Connector
App
+ Connector
App
+ Connector
App
+ Connector
App
+ Connector
MariaDB MariaDB
Cooperative
Monitoring
(v. 2.5)
Cooperative Monitoring and Locking

Distributed Cache between MaxScale
Servers
MaxScale 1 MaxScale 2
OR

Columnstore Orchestration VIA API
MaxScale
Columnstore
Primary
Columnstore
Replica 1
Columnstore
Replica 2
Legend
Normal Traffic
S3
API
dbroot1 dbroot2 dbroot3

MaxScale
Columnstore
Primary
Columnstore
Replica 1
Columnstore
Replica 2
Legend
Normal Traffic
S3
API
dbroot2 dbroot3dbroot1

MaxScale
Columnstore
Primary
Columnstore
Primary
Columnstore
Replica 2
Legend
Normal Traffic
S3
API
dbroot2 dbroot3dbroot1

THE PROBLEM OF BIG DATA
The analytic workload is not too complex
(compared to scientific work)...
… BUT AS DATA GROWS,
PERFORMANCE DEGRADES
MERELY BECAUSE OF ITS SIZE
Computers are not designed to
scale well at the hardware level…
… SO SCALING NEEDS TO BE DONE
AT THE SOFTWARE/ ALGORITHM
LEVEL

WHAT IS MARIADB COLUMNSTORE?
● Distributed and columnar storage for analytics
○ Stores data by column rather than row (file per column)
○ Stores data on separate, dedicated storage nodes
○ Scale out by adding more storage nodes
○ Can use GlusterFS or S3 for high availability
● Compression: up to 90%

● High performance storage engine
● Fully compatible with standard SQL
● Easier big data enterprise analytics
○ Single interface for OLTP and
analytics
○ Easy to integrate with third-party
tools
● Fast import of data (cpimport),
● Direct injection of data (bypassing SQL
layer)
○ No need for indexes and views
Features
● Powerful in analytics (Star/Snowflake
schemas)
● Scalability (just adding a new node(PM)
to the cluster and no data distribution is
required)
● Concurrency,
● File level backup in real-time
● Good compression -> low disk space
usage
● Faster, more efficient queries
○ Distributed parallel processing
WHY USE MARIADB COLUMNSTORE?

ROW-ORIENTED VERSUS COLUMN-ORIENTED
ROW-ORIENTED
● Rows stored
sequentially in a file
● Scans through every
record row by row
COLUMN-ORIENTED
● Each column is
stored in a separate file
● Scans only the
relevant column
ID FNAME LNAME STATE ZIP PHONE AGE GENDER
1 Bugs Bunny NY 11217 (718) 938-3235 34 M
2 Yosemite Sam CA 95389 (209) 375-6572 52 M
3 Daffy Duck NY 10013 (212) 227-1810 35 M
4 Elmer Fudd ME 04578 (207) 882-7323 43 M
5 Witch Hazel MA 01970 (978) 744-0991 57 F
ID FNAME LNAME STATE ZIP PHONE AGE GENDER
1 Bugs Bunny NY 11217 (718) 938-3235 34 M
2 Yosemite Sam CA 95389 (209) 375-6572 52 M
3 Daffy Duck NY 10013 (212) 227-1810 35 M
4 Elmer Fudd ME 04578 (207) 882-7323 43 M
5 Witch Hazel MA 01970 (978) 744-0991 57 F
> SELECT Fname FROM TABLE1 WHERE State = ‘NY’;

Storage Architecture Reduced I/O
High Performance Query Processing
● Only touch column files that are
in Filter, Projection, Group By,
and Join Conditions
● Eliminate Disk Block touches
to Partitions Outside Filter
and Join Conditions
Extent 1:
ShipDate: 2016-01-12 - 2016-03-05
Extent 2:
ShipDate: 2016-03-05 - 2016-09-23
Extent 3:
ShipDate: 2016-09-24 - 2017-01-06
> SELECT item, SUM(qty) FROM orders
WHERE ship_date BETWEEN '2016-01-01'
AND '2016-01-31' GROUP BY item
Horizontal
Partition:
8 Million Rows
Extent 2
Horizontal
Partition:
8 Million Rows
Extent 3
Horizontal
Partition:
8 Million Rows
Extent 1
id order_id Line item qty price supplier ship_date mode
1 1 1 Laptop 5 1000 Dell 2016-01-12 G
2 1 2 Monitor 5 200 LG 2016-01-13 G
3 2 1 Mouse 1 20 Logitech 2016-02-05 M
4 3 1 Laptop 3 1600 Apple 2016-01-31 P
... ... ... ... ... ... ... ... ...
8M 2016-03-05
8M+1 2016-03-05
... ... ... ... ... ... ... ... ...
16M 2016-09-23
16M+1 2016-09-24
... ... ... ... ... ... ... ... ...
24M 2017-01-06
ELIMINATED PARTITION
ELIMINATED PARTITION

Sample Two Node ColumnStore Cluster
App BI
MariaDB w/
ColumnStore
MaxScale
Legend
Normal Traffic
InnoDB BinLog
Replication
MariaDB w/
ColumnStore
Node 1 Node 2
ColumnStore Traffic

New Features in 1.5
● New API (Replaces OAM)
● PowerBI Direct Query Connector
● Improved Collation and Character
Sets
● Improved Write Cache (100x faster)
Phase 1 (Summer 2020)
● Faster Replication From Innodb
● Sequences
● Restart of Broken LDI (Checkpoints)
Phase 3 (Winter 2020)
● Multinode in SkySQL
● Wider Extents
● Extended Oracle Mode
● Dec(38)
Phase 2 (Fall 2020)
● Extent Elimination for DOUBLE
● Improved redistributedata
● Dynamic Columns
Phase 4 (Spring 2021)

NEW INSTALL PROCEDURE
[centos@localhost ~]$ wget https://dlm.mariadb.com/enterprise-release-helpers/mariadb_es_repo_setup
[centos@localhost ~]$ chmod +x mariadb_es_repo_setup
[centos@localhost ~]$ mariadb_es_repo_setup --token=<YOUR_TOKEN_HERE> --apply
[centos@localhost ~]$ yum -y install MariaDB-server MariaDB-columnstore-engine
[centos@localhost ~]$ systemctl enable mariadb-columnstore
[centos@localhost ~]$ systemctl start mariadb-columnstore
Before starting Columnstore, you might want to configure StorageManager and/or other
Columnstore variables. We will discuss later in this presentation.

Configure Columnstore
We provide a couple Columnstore.xml utilities.
mcsGetConfig -a
Will show the current Columnstore.xml settings.
To change a variable, run:
mcsSetConfig <group> <parameter> <value>
Example: mcsSetConfig CrossEngineSupport User “myuser”
Note: A restart of Columnstore is required.

Redundant File System (S3)
App BI
MariaDB w/
ColumnStore
MaxScale
Legend
Normal Traffic
InnoDB BinLog
Replication
MariaDB w/
ColumnStore
ColumnStore Storage Layer
Node 1 Node 2

To Configure Storage Manager
Edit this file /etc/columnstore/storagemanager.cnf:
[ObjectStorage]
…
service = S3
…
[S3]
bucket = yourbucketname
endpoint = s3.amazonaws.com
aws_access_key_id = <youraccesskeyid>
aws_secret_access_key= <yoursecretkey>

What to expect from MariaDB Platform X5, part 2

Recomendados

Recomendados

Mais conteúdo relacionado

Mais procurados

Mais procurados (20)

Semelhante a What to expect from MariaDB Platform X5, part 2

Semelhante a What to expect from MariaDB Platform X5, part 2 (20)

Mais de MariaDB plc

Mais de MariaDB plc (20)

Último

Último (20)

What to expect from MariaDB Platform X5, part 2