More Related Content Similar to P90 X Your Database!! (20) More from Denish Patel (16) P90 X Your Database!!2. Your Database!!
Denish Patel
Database Architect
OmniTi Computer Consulting Inc.
denish@omniti.com
1
Friday, March 25, 2011
4. Who am I ?
• Denish Patel
• Database Architect
• 4+ Years with OmniTi
• Expertise
• Heterogeneous Databases
• PostgreSQL, Oracle, MySQL
• Very Large databases
• Performance tuning
• Application level
• SQL tuning
• Database and OS parameters
• Objects tuning 2
Friday, March 25, 2011
5. Who am I ?
• Denish Patel
• Database Architect
• 4+ Years with OmniTi
• Expertise
• Heterogeneous Databases
• PostgreSQL, Oracle, MySQL
• Very Large databases
• Performance tuning
• Application level OmniTi is hiring!
• SQL tuning
• Database and OS parameters
• Objects tuning 2
Friday, March 25, 2011
7. Agenda
•Why bloat ?
•Issues
•Identify bloat
•Tables
•Indexes
•Bloat removal toolkit
•Monitoring and trending
3
Friday, March 25, 2011
9. why bloat?
•MVCC
•ACID
•Updates & Deletes
4
Friday, March 25, 2011
13. MVCC method-1
!
34..-+56''
' !:723,'
7$89$"&'
!"#$%&'
,+-.$'
/+&+' (0/1,2'
()*+&$ ><??!,'
'
/2;2,2'
72;2>,''
,31:7'
;<='
5
Friday, March 25, 2011
17. MVCC method-2
!
& 3./,-!&
!"#$%&
/,0,4!&& '"("&
)*'+!,&
!-+./& ',0,!,&
012&
41553!&
6
Friday, March 25, 2011
19. Issues
•Unnecessary costly I/O operations
•Poor database tables access time
•Increase maintenance time
•Waste of resources
•RAM
•CPU
•Disk
7
Friday, March 25, 2011
21. Prevent Bloat
• autovacuum =on
• vacuum manually between batch updates
• Truncate table
• Free Space Map
• <= PostgreSQL 8.3
• max_fsm_pages
• max_fsm_tables
• >=PostgreSQL 8.4
• auto sizing free space map
8
Friday, March 25, 2011
23. BEWARE!
• Bug: long running auto-vacuum workers
• PostgreSQL Version Affected
• 8.3.10
• 8.3.11
• 8.3.12
• 8.4.4
• 8.4.5
9
Friday, March 25, 2011
24. BEWARE!
• Bug: long running auto-vacuum workers
• PostgreSQL Version Affected
• 8.3.10
• 8.3.11
• 8.3.12
• 8.4.4
• 8.4.5
Upgrade to latest release !!
9
Friday, March 25, 2011
26. Measure Bloat
•Bloat Report
•pg_bloat_report (OmniTI Labs)
•Nagios Monitoring
•Check_postgres_bloat
10
Friday, March 25, 2011
28. pg_bloat_report
Bloat Report Tables
1. omniti.hits 1969348 of 4255960 pages wasted (46.3%),
15 GB of 32 GB.
2. omniti.clients 279979 of 370509 pages wasted (75.6%),
2187 MB of 2895 MB.
Bloat Report Indexes
1. omniti.hits_pk 51275 of 71075 pages wasted (72.1%),
401 MB of 555 MB.
2. omniti.clients_tz 47550 of 71271 pages wasted (66.7%),
371 MB 11 557 MB.
of
Friday, March 25, 2011
30. check_postgres_bloat
Notification Type: PROBLEM
Host: omniti
State: CRITICAL
Date/Time: Wed Feb 16 11:24:56 UTC 2011
Additional Info: POSTGRES_BLOAT
CRITICAL: DB "omniti" (host:omniti)
(port=5432) table omniti.hits rows:23995164
pages:2858893 shouldbe:2278176 (1.3X)
wasted size:4757233664 (4 GB)
12
Friday, March 25, 2011
32. bloat removal
toolkit
• Offline - with locking
• Vacuum
• Vacuum Full
• Cluster
• Cluster vs Vacuum Full
• Online - with minimal locking
• compact_table
• pg_reorg
13
Friday, March 25, 2011
36. vacuum
• Auto vacuum
• VACUUM [verbose|analyze] table
• Reclaims storage occupied by dead
tuples
• Reclaimed space available to re-use
• Normally free space doesn’t return to
OS
• No exclusive table lock
15
Friday, March 25, 2011
38. vacuum full
• VACUUM FULL table
• Compact table based on dead rows
• Starting PostgreSQL 9.0 , it’s rewrite
entire table and indexes (like CLUSTER)
• Reclaimed space available to re-use
• Space returned to OS
• Table Exclusive lock
• Expensive operation
16
Friday, March 25, 2011
42. cluster
•CLUSTER [verbose] table_name
[USING index_name]
•Create reorganized copy of table
based on index
•Rebuild indexes too
•Requires Exclusive Lock on table
•Expensive Operation
18
Friday, March 25, 2011
44. cluster vs vacuum full
It’s just rewrite of table without
Order based on Index
any specific order
Table needs index Doesn’t need index
Bloat indexes (Pre PostgreSQL
Doesn’t bloat index
9.0)
Overall Operation Slower (Pre
Overall operation Faster
PostgreSQL 9.0)
Recommended Not recommended
19
Friday, March 25, 2011
48. compact_table
•Developed by OmniTi
•Reorganize rows to empty pages at the
end based on ctid
•ctid - (page number, tuple number)
•When we update a row, the row's ctid
changes, because the update creates a
new version of the row and leaves the
old version behind
21
Friday, March 25, 2011
50. compact_table
postgres=# select ctid,* from bar;
ctid | a | b
-------+---+---
(0,1) | 1 | 1
(0,2) | 2 | 1
(2 rows)
postgres=# update bar set a=3 where a=2;
UPDATE 1
postgres=# select ctid,* from bar;
ctid | a | b
-------+---+---
(0,1) | 1 | 1
(0,3) | 3 | 1
(2 rows)
postgres=# vacuum verbose bar;
.
.
INFO: "bar": found 1 removable, 2 nonremovable row versions in 1 out of 1 pages
DETAIL: 0 dead row versions cannot be removed yet.
VACUUM
22
Friday, March 25, 2011
53. compact_table
postgres=# truncate table bar;
TRUNCATE TABLE
postgres=# insert into bar select generate_series(1,1000);
INSERT 0 1000
postgres=# delete from bar where a %2 = 0;
DELETE 500
postgres=# select max(ctid) from bar;
max
--------
(4,95)
(1 row)
postgres=# vacuum verbose bar;
INFO: "bar": found 0 removable, 500
nonremovable row versions in 5 out of 5 pages
VACUUM
23
Friday, March 25, 2011
55. compact_table
postgres=# begin;
BEGIN
postgres=# update bar set a=a where ctid>='(3,0)';
UPDATE 161
postgres=# update bar set a=a where ctid>='(3,0)';
UPDATE 161
postgres=# update bar set a=a where ctid>='(3,0)';
UPDATE 48
postgres=# update bar set a=a where ctid>='(3,0)';
UPDATE 48
postgres=# update bar set a=a where ctid>='(3,0)';
UPDATE 34
postgres=# update bar set a=a where ctid>='(3,0)';
UPDATE 0
postgres=# commit;
COMMIT
postgres=# vacuum verbose bar;
.
.
INFO: "bar": truncated 5 to 3 pages
.
VACUUM
24
Friday, March 25, 2011
57. compact_table
•Pretty safe update process
•No additional changes required
•Slow process
•Generate Index bloat
•Under development
•Not tested in production
•Future consideration?
25
Friday, March 25, 2011
61. pg_reorg
• Developed by NTT OSS Center
• Reorganize table with very short lock
• Needs Primary Key
• Not Null Unique Key works!
• CLUSTER online
• ORDER BY Key
• VACUUM FULL Online
• Rewrite without ORDER BY
• Build and Install like other Contrib Module
27
Friday, March 25, 2011
62. !
3,4-$*(
8,$%"&-/(
9"#:(($-./*(
<%$=(
!"#$%&'(
;-#'*$( 8,*#-$%"&-/(
9"#:( )"'(
;-./*(
;-./*(
(
6*,/-7(
;*+,(
)"'((
;-./*(
)"'($"($*+,(
$-./*0( !<-,,%&'(
( 1&2*#$(
( 3,4-$*(
<%$=(>*#7(
( 5*/*$*( 2="#$()"?:(
28
Friday, March 25, 2011
63. !
pg_reorg
3,4-$*(
8,$%"&-/(
9"#:(($-./*(
<%$=(
!"#$%&'(
;-#'*$( 8,*#-$%"&-/(
9"#:( )"'(
;-./*(
;-./*(
(
6*,/-7(
;*+,(
)"'((
;-./*(
)"'($"($*+,(
$-./*0( !<-,,%&'(
( 1&2*#$(
( 3,4-$*(
<%$=(>*#7(
( 5*/*$*( 2="#$()"?:(
28
Friday, March 25, 2011
65. pg_reorg Options
pg_reorg --no-order -table test -d omniti
-n
-- no-order
Do online VACUUM FULL.
-o columns [,...]
--order-by=columns [,...]
Do online CLUSTER ordered by specified columns.
-t table
--table=table
-Z
--no-analyze
-T seconds
--wait-timeout=seconds
29
Friday, March 25, 2011
67. pg_reorg in action
$ /opt/pgsql/bin/pg_reorg -e -t public.test --order-by=test_id -U postgres -d test
LOG: (query) SET statement_timeout = 0
LOG: (query) SET search_path = pg_catalog, pg_temp, public
LOG: (query) SET client_min_messages = warning
LOG: (query) SELECT * FROM reorg.tables WHERE relid = $1::regclass
LOG: (param:0) = public.test
LOG: (query) BEGIN ISOLATION LEVEL READ COMMITTED
LOG: (query) SET LOCAL statement_timeout = 100
LOG: (query) LOCK TABLE test IN ACCESS EXCLUSIVE MODE
LOG: (query) RESET statement_timeout
LOG: (query) SELECT reorg.conflicted_triggers($1)
LOG: (param:0) = 7441636
LOG: (query) CREATE TYPE reorg.pk_7441636 AS (test_id integer)
LOG: (query) CREATE TABLE reorg.log_7441636 (id bigserial PRIMARY KEY, pk
reorg.pk_7441636, row pulbic.test)
LOG: (query) CREATE TRIGGER z_reorg_trigger BEFORE INSERT OR DELETE OR
UPDATE ON pulbic.test FOR EACH ROW EXECUTE PROCEDURE
reorg.reorg_trigger('INSERT INTO reorg.log_7441636(pk, row) VALUES( CASE
WHEN $1 IS NULL THEN NULL ELSE (ROW($1.test_id)::reorg.pk_7441636) END,
$2)')
LOG: (query) SELECT reorg.disable_autovacuum('reorg.log_7441636')
LOG: (query) COMMIT
30
Friday, March 25, 2011
68. pg_reorg in action
$ /opt/pgsql/bin/pg_reorg -e -t public.test --order-by=test_id -U postgres -d test
LOG: (query) SET statement_timeout = 0
LOG: (query) SET search_path = pg_catalog, pg_temp, public SET
LOG: (query) SET client_min_messages = warning
LOG: (query) SELECT * FROM reorg.tables WHERE relid = $1::regclass
variables
LOG: (param:0) = public.test
LOG: (query) BEGIN ISOLATION LEVEL READ COMMITTED
LOG: (query) SET LOCAL statement_timeout = 100
LOG: (query) LOCK TABLE test IN ACCESS EXCLUSIVE MODE
LOG: (query) RESET statement_timeout
LOG: (query) SELECT reorg.conflicted_triggers($1)
LOG: (param:0) = 7441636
LOG: (query) CREATE TYPE reorg.pk_7441636 AS (test_id integer)
LOG: (query) CREATE TABLE reorg.log_7441636 (id bigserial PRIMARY KEY, pk
reorg.pk_7441636, row pulbic.test)
LOG: (query) CREATE TRIGGER z_reorg_trigger BEFORE INSERT OR DELETE OR
UPDATE ON pulbic.test FOR EACH ROW EXECUTE PROCEDURE
reorg.reorg_trigger('INSERT INTO reorg.log_7441636(pk, row) VALUES( CASE
WHEN $1 IS NULL THEN NULL ELSE (ROW($1.test_id)::reorg.pk_7441636) END,
$2)')
LOG: (query) SELECT reorg.disable_autovacuum('reorg.log_7441636')
LOG: (query) COMMIT
30
Friday, March 25, 2011
69. pg_reorg in action
$ /opt/pgsql/bin/pg_reorg -e -t public.test --order-by=test_id -U postgres -d test
LOG: (query) SET statement_timeout = 0
LOG: (query) SET search_path = pg_catalog, pg_temp, public SET
LOG: (query) SET client_min_messages = warning
LOG: (query) SELECT * FROM reorg.tables WHERE relid = $1::regclass
variables
LOG: (param:0) = public.test
LOG: (query) BEGIN ISOLATION LEVEL READ COMMITTED
LOG: (query) SET LOCAL statement_timeout = 100
LOG: (query) LOCK TABLE test IN ACCESS EXCLUSIVE MODE
LOG: (query) RESET statement_timeout
LOG: (query) SELECT reorg.conflicted_triggers($1)
Place Triggers
LOG: (param:0) = 7441636
LOG: (query) CREATE TYPE reorg.pk_7441636 AS (test_id integer)
LOG: (query) CREATE TABLE reorg.log_7441636 (id bigserial PRIMARY KEY, pk
reorg.pk_7441636, row pulbic.test)
LOG: (query) CREATE TRIGGER z_reorg_trigger BEFORE INSERT OR DELETE OR
UPDATE ON pulbic.test FOR EACH ROW EXECUTE PROCEDURE
reorg.reorg_trigger('INSERT INTO reorg.log_7441636(pk, row) VALUES( CASE
WHEN $1 IS NULL THEN NULL ELSE (ROW($1.test_id)::reorg.pk_7441636) END,
$2)')
LOG: (query) SELECT reorg.disable_autovacuum('reorg.log_7441636')
LOG: (query) COMMIT
30
Friday, March 25, 2011
71. pg_reorg in action
LOG: (query) BEGIN ISOLATION LEVEL SERIALIZABLE
LOG: (query) SELECT set_config('work_mem', current_setting('maintenance_work_mem'),
true)
LOG: (query) SELECT reorg.array_accum(virtualtransaction) FROM pg_locks WHERE
locktype = 'virtualxid' AND pid <> pg_backend_pid()
LOG: (query) DELETE FROM reorg.log_7441636
LOG: (query) CREATE TABLE reorg.table_7441636 WITH (oids=false) TABLESPACE
pg_default AS SELECT * FROM ONLY public.test ORDER BY test_id
LOG: (query) SELECT reorg.disable_autovacuum('reorg.table_7441636')
LOG: (query) COMMIT
LOG: (query) SELECT indexrelid, reorg.reorg_indexdef(indexrelid, indrelid) FROM
pg_index WHERE indrelid = $1
LOG: (param:0) = 7441636
LOG: (query) CREATE UNIQUE INDEX index_201337719 ON reorg.table_7441636 USING
btree (test_id)
LOG: (query) SELECT reorg.reorg_apply($1, $2, $3, $4, $5, $6)
LOG: (param:0) = SELECT * FROM reorg.log_7441636 ORDER BY id LIMIT $1
LOG: (param:1) = INSERT INTO reorg.table_7441636 VALUES ($1.*)
LOG: (param:2) = DELETE FROM reorg.table_7441636 WHERE (test_id) = ($1.test_id)
LOG: (param:3) = UPDATE reorg.table_7441636 SET (test_id, name, lat, lon, city, country,
state_abr) = ($2.test_id, $2.name, $2.lat, $2.lon, $2.city, $2.country, $2.state_abr) WHERE
(test_id) = ($1.test_id)
LOG: (param:4) = DELETE FROM reorg.log_7441636 WHERE id <= $1
LOG: (param:5) = 1000 31
Friday, March 25, 2011
72. pg_reorg in action
LOG: (query) BEGIN ISOLATION LEVEL SERIALIZABLE
LOG: (query) SELECT set_config('work_mem', current_setting('maintenance_work_mem'),
true)
LOG: (query) SELECT reorg.array_accum(virtualtransaction) FROM pg_locks WHERE
locktype = 'virtualxid' AND pid <> pg_backend_pid() CTAS
LOG: (query) DELETE FROM reorg.log_7441636 Create
LOG: (query) CREATE TABLE reorg.table_7441636 WITH (oids=false) TABLESPACE
pg_default AS SELECT * FROM ONLY public.test ORDER BY test_id Indexes
LOG: (query) SELECT reorg.disable_autovacuum('reorg.table_7441636')
LOG: (query) COMMIT
LOG: (query) SELECT indexrelid, reorg.reorg_indexdef(indexrelid, indrelid) FROM
pg_index WHERE indrelid = $1
LOG: (param:0) = 7441636
LOG: (query) CREATE UNIQUE INDEX index_201337719 ON reorg.table_7441636 USING
btree (test_id)
LOG: (query) SELECT reorg.reorg_apply($1, $2, $3, $4, $5, $6)
LOG: (param:0) = SELECT * FROM reorg.log_7441636 ORDER BY id LIMIT $1
LOG: (param:1) = INSERT INTO reorg.table_7441636 VALUES ($1.*)
LOG: (param:2) = DELETE FROM reorg.table_7441636 WHERE (test_id) = ($1.test_id)
LOG: (param:3) = UPDATE reorg.table_7441636 SET (test_id, name, lat, lon, city, country,
state_abr) = ($2.test_id, $2.name, $2.lat, $2.lon, $2.city, $2.country, $2.state_abr) WHERE
(test_id) = ($1.test_id)
LOG: (param:4) = DELETE FROM reorg.log_7441636 WHERE id <= $1
LOG: (param:5) = 1000 31
Friday, March 25, 2011
73. pg_reorg in action
LOG: (query) BEGIN ISOLATION LEVEL SERIALIZABLE
LOG: (query) SELECT set_config('work_mem', current_setting('maintenance_work_mem'),
true)
LOG: (query) SELECT reorg.array_accum(virtualtransaction) FROM pg_locks WHERE
locktype = 'virtualxid' AND pid <> pg_backend_pid() CTAS
LOG: (query) DELETE FROM reorg.log_7441636 Create
LOG: (query) CREATE TABLE reorg.table_7441636 WITH (oids=false) TABLESPACE
pg_default AS SELECT * FROM ONLY public.test ORDER BY test_id Indexes
LOG: (query) SELECT reorg.disable_autovacuum('reorg.table_7441636')
LOG: (query) COMMIT
LOG: (query) SELECT indexrelid, reorg.reorg_indexdef(indexrelid, indrelid) FROM
pg_index WHERE indrelid = $1
LOG: (param:0) = 7441636
LOG: (query) CREATE UNIQUE INDEX index_201337719 ON reorg.table_7441636 USING
btree (test_id)
Apply
LOG: (query) SELECT reorg.reorg_apply($1, $2, $3, $4, $5, $6)
LOG: (param:0) = SELECT * FROM reorg.log_7441636 ORDER BY id LIMIT $1 Operational
LOG: (param:1) = INSERT INTO reorg.table_7441636 VALUES ($1.*) logs
LOG: (param:2) = DELETE FROM reorg.table_7441636 WHERE (test_id) = ($1.test_id)
LOG: (param:3) = UPDATE reorg.table_7441636 SET (test_id, name, lat, lon, city, country,
state_abr) = ($2.test_id, $2.name, $2.lat, $2.lon, $2.city, $2.country, $2.state_abr) WHERE
(test_id) = ($1.test_id)
LOG: (param:4) = DELETE FROM reorg.log_7441636 WHERE id <= $1
LOG: (param:5) = 1000 31
Friday, March 25, 2011
75. pg_reorg in action
LOG: (query) BEGIN ISOLATION LEVEL READ COMMITTED
LOG: (query) SET LOCAL statement_timeout = 100
LOG: (query) LOCK TABLE public.test IN ACCESS EXCLUSIVE MODE
LOG: (query) RESET statement_timeout
LOG: (query) SELECT reorg.reorg_apply($1, $2, $3, $4, $5, $6)
LOG: (param:0) = SELECT * FROM reorg.log_7441636 ORDER BY id LIMIT $1
LOG: (param:1) = INSERT INTO reorg.table_7441636 VALUES ($1.*)
LOG: (param:2) = DELETE FROM reorg.table_7441636 WHERE (test_id) = ($1.test_id)
LOG: (param:3) = UPDATE reorg.table_7441636 SET (test_id, name, lat, lon, city,
country, state_abr) = ($2.test_id, $2.name, $2.lat, $2.lon, $2.city, $2.country, $2.state_abr)
WHERE (test_id) = ($1.test_id)
LOG: (param:4) = DELETE FROM reorg.log_7441636 WHERE id <= $1
LOG: (param:5) = 0
LOG: (query) SELECT reorg.reorg_swap($1)
LOG: (param:0) = 7441636
LOG: (query) COMMIT
LOG: (query) BEGIN ISOLATION LEVEL READ COMMITTED
LOG: (query) SELECT reorg.reorg_drop($1)
LOG: (param:0) = 7441636
LOG: (query) COMMIT
LOG: (query) BEGIN ISOLATION LEVEL READ COMMITTED
LOG: (query) ANALYZE publoc.test
32
LOG: (query) COMMIT
Friday, March 25, 2011
76. pg_reorg in action
LOG: (query) BEGIN ISOLATION LEVEL READ COMMITTED Lock
LOG: (query) SET LOCAL statement_timeout = 100 table and
LOG: (query) LOCK TABLE public.test IN ACCESS EXCLUSIVE MODE apply
LOG: (query) RESET statement_timeout
LOG: (query) SELECT reorg.reorg_apply($1, $2, $3, $4, $5, $6) operational logs
LOG: (param:0) = SELECT * FROM reorg.log_7441636 ORDER BY id LIMIT $1 last time
LOG: (param:1) = INSERT INTO reorg.table_7441636 VALUES ($1.*)
LOG: (param:2) = DELETE FROM reorg.table_7441636 WHERE (test_id) = ($1.test_id)
LOG: (param:3) = UPDATE reorg.table_7441636 SET (test_id, name, lat, lon, city,
country, state_abr) = ($2.test_id, $2.name, $2.lat, $2.lon, $2.city, $2.country, $2.state_abr)
WHERE (test_id) = ($1.test_id)
LOG: (param:4) = DELETE FROM reorg.log_7441636 WHERE id <= $1
LOG: (param:5) = 0
LOG: (query) SELECT reorg.reorg_swap($1)
LOG: (param:0) = 7441636
LOG: (query) COMMIT
LOG: (query) BEGIN ISOLATION LEVEL READ COMMITTED
LOG: (query) SELECT reorg.reorg_drop($1)
LOG: (param:0) = 7441636
LOG: (query) COMMIT
LOG: (query) BEGIN ISOLATION LEVEL READ COMMITTED
LOG: (query) ANALYZE publoc.test
32
LOG: (query) COMMIT
Friday, March 25, 2011
77. pg_reorg in action
LOG: (query) BEGIN ISOLATION LEVEL READ COMMITTED Lock
LOG: (query) SET LOCAL statement_timeout = 100 table and
LOG: (query) LOCK TABLE public.test IN ACCESS EXCLUSIVE MODE apply
LOG: (query) RESET statement_timeout
LOG: (query) SELECT reorg.reorg_apply($1, $2, $3, $4, $5, $6) operational logs
LOG: (param:0) = SELECT * FROM reorg.log_7441636 ORDER BY id LIMIT $1 last time
LOG: (param:1) = INSERT INTO reorg.table_7441636 VALUES ($1.*)
LOG: (param:2) = DELETE FROM reorg.table_7441636 WHERE (test_id) = ($1.test_id)
LOG: (param:3) = UPDATE reorg.table_7441636 SET (test_id, name, lat, lon, city,
country, state_abr) = ($2.test_id, $2.name, $2.lat, $2.lon, $2.city, $2.country, $2.state_abr)
WHERE (test_id) = ($1.test_id)
LOG: (param:4) = DELETE FROM reorg.log_7441636 WHERE id <= $1
LOG: (param:5) = 0
LOG: (query) SELECT reorg.reorg_swap($1) Swapping
LOG: (param:0) = 7441636
LOG: (query) COMMIT
LOG: (query) BEGIN ISOLATION LEVEL READ COMMITTED
LOG: (query) SELECT reorg.reorg_drop($1)
LOG: (param:0) = 7441636
LOG: (query) COMMIT
LOG: (query) BEGIN ISOLATION LEVEL READ COMMITTED
LOG: (query) ANALYZE publoc.test
32
LOG: (query) COMMIT
Friday, March 25, 2011
78. pg_reorg in action
LOG: (query) BEGIN ISOLATION LEVEL READ COMMITTED Lock
LOG: (query) SET LOCAL statement_timeout = 100 table and
LOG: (query) LOCK TABLE public.test IN ACCESS EXCLUSIVE MODE apply
LOG: (query) RESET statement_timeout
LOG: (query) SELECT reorg.reorg_apply($1, $2, $3, $4, $5, $6) operational logs
LOG: (param:0) = SELECT * FROM reorg.log_7441636 ORDER BY id LIMIT $1 last time
LOG: (param:1) = INSERT INTO reorg.table_7441636 VALUES ($1.*)
LOG: (param:2) = DELETE FROM reorg.table_7441636 WHERE (test_id) = ($1.test_id)
LOG: (param:3) = UPDATE reorg.table_7441636 SET (test_id, name, lat, lon, city,
country, state_abr) = ($2.test_id, $2.name, $2.lat, $2.lon, $2.city, $2.country, $2.state_abr)
WHERE (test_id) = ($1.test_id)
LOG: (param:4) = DELETE FROM reorg.log_7441636 WHERE id <= $1
LOG: (param:5) = 0
LOG: (query) SELECT reorg.reorg_swap($1) Swapping
LOG: (param:0) = 7441636
LOG: (query) COMMIT
LOG: (query) BEGIN ISOLATION LEVEL READ COMMITTED
LOG: (query) SELECT reorg.reorg_drop($1) Drop work and
LOG: (param:0) = 7441636 temp tables
LOG: (query) COMMIT
LOG: (query) BEGIN ISOLATION LEVEL READ COMMITTED
Analyze new table
LOG: (query) ANALYZE publoc.test
32
LOG: (query) COMMIT
Friday, March 25, 2011
80. pg_reorg in
Production
•Database Size 540GB
•OLTP database 2-3K transactions/sec
•Largest pre-rebuild table Size was
127GB
•Took 5.75 hours to rebuild
•Reclaimed 52GB
•No outage reported for website!
33
Friday, March 25, 2011
82. pg_reorg in
Production
Table Size Before Pg_Reorg
Bloat Removed
Reorg Execution time
35GB 24m 10sec 13GB
27GB 116m 8 Sec 6GB
31GB 64m 32sec 4GB
44GB 128m 44sec 3GB
34
Friday, March 25, 2011
88. Test Case
• Database : PostgreSQL 9.0.2
• OS: Solaris 11
• RAM : 32GB
postgres=# d users
• work_mem=16MB Table "public.users"
Column | Type | Modifiers
• Table Name : Users --------------+--------------------------+-----------
user_id | integer | not null
• Table Size : 1180 MB name | text |
creation_tsz | timestamp with time zone |
• Indexes Size : 256 MB active | boolean |
clicks | double precision |
• Rows : 6M Indexes:
"users_pkey" PRIMARY KEY, btree (user_id)
"users_creation_tsz" btree (creation_tsz)
- DELETE 3M ROWS
37 - UPDATE 1M ROWS
Friday, March 25, 2011
90. Comparison
CLUSTER VACUUM FULL PG_REORG
Execution Time: Execution Time: Execution Time:
2 min 1 min 50 Sec 1 min 4 Sec
Table Size: 594MB Table Size: 594MB Table Size: 594MB
Index Size: 128 MB Index Size: 128 MB Index Size: 128 MB
offline offline online
Built - in Built-in External Tool
38
Friday, March 25, 2011
92. Other Tips
•Remove unused indexes
•Reindex indexes as needed
•Partitioned table
•Detach child table -> Cluster child
table -> Attach
•Avoid vacuum full (Pre PostgrSQL 9.0)
39
Friday, March 25, 2011
94. Monitoring
•Circonus
•Nagios monitoring
•Bloat report
40
Friday, March 25, 2011
96. trending
•Cacti
•Circonus
41
Friday, March 25, 2011
101. thanks!
•OmniTi
•Robert Treat
•Depesz
•Joshua (End Point)
•PGEast Conference Committee
43
Friday, March 25, 2011
103. References
•http://www.postgresql.org/docs/9.0/
•http://reorg.projects.postgresql.org/
pg_reorg.html
•https://labs.omniti.com/pgtreats/trunk/
tools/compact_table
•http://blog.endpoint.com/2010/09/
reducing-bloat-without-locking.html
44
Friday, March 25, 2011
105. open discussion
•When PostgreSQL will have built in
features to rebuild table and index
online ?
•Is it worth to change MVCC handling
to avoid bloating?
45
Friday, March 25, 2011
107. Questions?
•Email : denish@omniti.com
•Twitter : http://twitter.com/#!/denishpatel
46
Friday, March 25, 2011