1. Leveraging In-Memory Storage to
Overcome Oracle PGA Memory Limits
March 5, 2014
Presented by: Alex Fatkulin
Senior Consultant

2. Who am I ?
 Senior Technical Consultant at Enkitec
 12 years using Oracle
 Clustered and HA solutions
 Database Development and Design
 Technical Reviewer
 Blog at http://afatkulin.blogspot.com
3

3. Why This Presentation?
4

4. Data Growth and Processing
 Data Volumes UP
 Processing Power UP
 Database Design (in general) DOWN
5
Data Volume
Processing Power
Database Design
BetterWorse

5. 6
How many people have systems with
more than 128G of RAM?

6. “640K” Problems
7
 Processing patterns change
 Things that didn’t matter before matter now
 Legacy RDBMS code vs 64-bit systems
 KIWI (Kill It With Iron)

7. PGA Memory
8

8. PGA Memory (Dedicated Server)
9
Sort AreaHash Area Bitmap Area
SQL Work Areas
Session Memory Private SQL Area
Process Memory

9. Query Execution Work Areas
10
Work Area Size
Manual Management Auto Management
hash_area_size
sort_area_size
bitmap_merge_area_size
create_bitmap_area_size
pga_aggregate_target

10. PGA Memory Limits
11

11. Manual Work Area Management
12
SQL> alter session set workarea_size_policy=manual;
Session altered
SQL> alter session set hash_area_size=4294967296; -- 4GB
alter session set hash_area_size=4294967296
ORA-02017: integer value required
 11.2.0.4 Linux x64
SQL> alter session set hash_area_size=2147483648; -- 2GB
alter session set hash_area_size=2147483648
ORA-02017: integer value required
SQL> alter session set hash_area_size=2147483647; -- 2GB - 1
Session altered
 32-bit Signed Integer Limit

12. Auto Work Area Management
13
SQL> alter system set pga_aggregate_target=1024g; -- 1TB
System altered
 11.2.0.4 Linux x64
 Where is the catch?
 PGA_AGGREGATE_TARGET
 _smm_max_size/_smm_max_size_static
 Maximum work area size per process (px/serial)
 _smm_px_max_size/_smm_px_max_size_static
 Maximum work area size per query (px)
 _pga_max_size
 Maximum PGA size per process (px/serial)

13. Auto Work Area Management
14
 PGA_AGGREGATE_TARGET 16M – 512M
0
100
200
300
400
500
600
16 32 64 128 256 512
pga_aggregate_target _smm_max_size _smm_px_max_size _pga_max_size
 _pga_max_size = 200M
 _smm_max_size[_static] = 20%
 _smm_px_max_size[_static] = 50%

14. Auto Work Area Management
15
 PGA_AGGREGATE_TARGET 1G – 10G
 _pga_max_size = 20%
 _smm_max_size[_static] = 10%
 _smm_px_max_size[_static] = 50%
0
2000
4000
6000
8000
10000
12000
pga_aggregate_target _smm_max_size _smm_px_max_size _pga_max_size

15. Auto Work Area Management
16
 PGA_AGGREGATE_TARGET 10G – 16G
 _pga_max_size = 2G
 _smm_max_size[_static] = 2G
 _smm_px_max_size[_static] = 50%
0
2000
4000
6000
8000
10000
12000
14000
16000
18000
10240 10752 11264 11776 12288 12800 13312 13824 14336 14848 15360 15872 16384
pga_aggregate_target _smm_max_size _smm_px_max_size _pga_max_size

16. Auto Work Area Management
17
 PGA_AGGREGATE_TARGET >= 10G
 _smm_max_size[_static] = 1G (maximum value)
 _pga_max_size = 2G (maximum value)
 If you’re bumping into per process limits further rising
pga_aggregate_target will not help

17. Per Process Limit
18
 PGA_AGGREGATE_TARGET = 192G / DOP = 16
 PGA_AGGREGATE_TARGET = 512G / DOP = 16

18. Per Process Limit
19
 PGA_AGGREGATE_TARGET = 192G / DOP = 16
 PGA_AGGREGATE_TARGET = 512G / DOP = 16
 _pga_max_size * DOP

19. Run With Higher DOP?
20
 Exadata X3-8
 2TB RAM (per node)
 80 CPU cores (per node)
 PGA_AGGREGATE_TARGET = 1536G
 Would require at least DOP 768 (_pga_max_size)
 9.6x core count
 Concurrency issues
 Manageability issues
 PX data distribution/algorithm issues

20. Run With Higher DOP?
21
 DOP vs CPU Cores Used
5 (spill) 8 (spill)
64 (no spill)
0
16
32
48
64
16 32 64
CPUCoresUsed
DOP

21. Run With Higher DOP?
22
 PX Algorithm Issues (ex.: median function)

22. Play With Underscores?
23
 MOS Doc ID 453540.1
 Allows _smm_max_size=2097151
 Allows _pga_max_size > 2GB with patch 17951233
 Can get 4G per process limit
 Can get > 4G per process limit
 /proc/sys/vm/max_map_count (OS page count)
 _realfree_heap_pagesize_hint (allocator page size)
 Weird behavior and likely not fully supported for work
areas (MOS Doc ID 453540.1)

23. Radically Improve TEMP I/O?
24
 TEMP I/O (in general)
 Doesn’t need redundancy
 Doesn’t need persistency
 Doesn’t need recoverability
 In-Memory FS (tmpfs, etc.)
 SAN/NAS LUN with write-back cache

24. Linux tmpfs (via loop device)
25
 10.1B rows / 416G HCC (QUERY HIGH)
SELECT /*+ parallel(t,16) */
CUST_ID, DATE_ID, COUNT(DISTINCT STORE_ID) D
FROM TRANS T
WHERE DATE_ID between to_date('01012012', 'ddmmyyyy') and to_date('31122013', 'ddmmyyyy')
GROUP BY CUST_ID, DATE_ID;
Exadata X3-8 TEMP
tmpfs TEMP
 13.8x faster TEMP I/O (237G)

25. Linux tmpfs (via loop device)
26
 10.1B rows / 416G HCC (QUERY HIGH)
SELECT /*+ parallel(t,16) */
CUST_ID, DATE_ID, COUNT(DISTINCT STORE_ID) D
FROM TRANS T
WHERE DATE_ID between to_date('01012012', 'ddmmyyyy') and to_date('31122013', 'ddmmyyyy')
GROUP BY CUST_ID, DATE_ID;

26. ZFSSA (Infiniband SRP)
27
 7.3B rows / 300G HCC (QUERY HIGH)
SELECT /*+ parallel(t,32) */
CUST_ID, DATE_ID, COUNT(DISTINCT STORE_ID) D
FROM TRANS T
GROUP BY CUST_ID, DATE_ID;
Exadata X3-8 TEMP
ZFSSA TEMP
(write-back cache)
 64.3x faster TEMP I/O (141G)

27. ZFSSA (Infiniband SRP)
28
 7.3B rows / 300G HCC (QUERY HIGH)
SELECT /*+ parallel(t,32) */
CUST_ID, DATE_ID, COUNT(DISTINCT STORE_ID) D
FROM TRANS T
GROUP BY CUST_ID, DATE_ID;

28. Summary
29
 Remember about per process limits
 Larger PGA_AGGREGATE_TARGET may do nothing
 Balance PGA/DOP/CPU
 It’s possible to increase TEMP I/O performance by 10x-
50x and more
 Oracle PGA code does not fully embrace 64-bit systems
at the moment

29. Q & A
Email: afatkulin@enkitec.com
Blog: http://afatkulin.blogspot.com
30