Mais conteúdo relacionado Semelhante a APEX Connect 2019 - array/bulk processing in PLSQL (20) Mais de Connor McDonald (19) APEX Connect 2019 - array/bulk processing in PLSQL1. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.
Connor McDonald
6. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.
6https://asktom.oracle.com
7. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.
https://asktom.oracle.com/officehours
8. Copyright © 2017, Oracle and/or its affiliates. All rights reserved.
200 hours free access so far
8
9. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.
before we begin
10. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.
let's look at real life ...
11. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.
"We'd never do that .... that's stupid"
12. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.
but everyone does ... :-(
13. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.
the most common PLSQL program
14. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.
SQL> declare
2 cursor c_products is
3 select aisle, item, descr
4 from hardware;
5 l_each c_products%rowtype;
6 begin
7 open c_products;
8 loop
9 fetch c_products
10 into l_each;
11 exit when c_products%notfound;
12 end loop;
13 close c_products;
14 end;
15 /
PL/SQL procedure successfully completed.
15. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.
our setup for this session
16. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.
SQL> create table HARDWARE
2 ( aisle NUMBER(5),
3 item NUMBER(12),
4 descr CHAR(50),
5 stocked DATE
6 );
Table created.
SQL> insert /*+ APPEND */ into HARDWARE
2 select trunc(rownum/1000)+1 aisle,
3 rownum item,
4 'Description '||rownum descr,
5 to_date('01-JAN-2011','dd-mon-yyyy')+rownum/86400
6 from
7 ( select 1 from dual connect by level <= 1000),
8 ( select 1 from dual connect by level <= 1000);
1000000 rows created.
SQL> create index HARDWARE_IX on HARDWARE ( aisle, item );
Index created.
1,000,000 items,
across 1000 aisles
quick access to
items in an aisle
17. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.
no matter what platform
19. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.
no matter what direction
21. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.
row by row :-(
22. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.
"The database is slow"
29. SQL> select *
2 from EMP
3 where …
SQL> update EMP
2 set …
3 where …
SQL> select *
2 from EMP
3 where …
SQL> insert into
2 EMP
3 values (…)
SQL> select *
2 from CUSTOMER
3 /
SQL> select max(EMP)
2 from DEPT
SQL> delete
2 from DEPT
3 /
46. spinning
can I have the latch ?
can I have the latch ?
can I have the latch ?
can I have the latch ?
can I have the latch ?
can I have the latch ?
can I have the latch ?
can I have the latch ?
can I have the latch ?
can I have the latch ?
can I have the latch ?
can I have the latch ?
can I have the latch ?
can I have the latch ?
can I have the latch ?
can I have the latch ?
can I have the latch ?
can I have the latch ?
can I have the latch ?
can I have the latch ?
can I have the latch ?
can I have the latch ?
can I have the latch ?
can I have the latch ?
can I have the latch ?
can I have the latch ?
can I have the latch ?
can I have the latch ?
46
58. • Get latch
• Search along list of blocks in memory
• Read block
• Extract row of interest
• Release latch
• Give row back to client
"fetch a row"
58
SQL>
...
9 fetch c_products
10 into l_each;
...
62. • Get latch
• Walk along list
• Get block
• Extract single row
• Release latch
• Give row back to client
• Get latch (again)
• Walk along list (again)
• Get block (again)
• Extract single row
• Release latch (again)
• Give row back to client (again)
Row #1
Row #2
• Get latch (again)
• Walk along list (again)
• Get block (again)
• Extract single row
• Release latch (again)
• Give row back to client (again)
Row #3
62
loop
fetch c_products
into l_each;
exit when c_products%notfound;
end loop;
64. "fetch a row"
• Get latch
• Walk along list
• Get block
• Pin the block
• Release latch
• Give row 1 back to client
• Give row 2 back to client
…
• Give row n to client
• Get latch
• Walk along list
• Remove my pin on the block
• Release latch
“and .. I’ll will need
several rows from
this block”
Connor is
using this
block
66. the client is in charge
SQL> set arraysize 50 stmt.setFetchSize(50)
proc prefetch=50 reader.FetchSize = 50
68. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.
quick revision
68
69. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.
implicit cursor via select
69
SQL> declare
2 l_descr hardware.descr%type;
3 begin
4 select descr
5 into l_descr
6 from hardware
7 where aisle = 1 -- single
8 and item = 1; -- row
9 end;
bulk
SQL> declare
2 type t_descr_list is
3 table of hardware.descr%type;
4 l_descr_list t_descr_list;
5 begin
6 select descr
7 bulk collect
8 into l_descr_list
9 from hardware
10 where aisle = 1 -- multiple
11 and item between 1 and 100; -- rows
12 end;
70. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.
explicit cursor
70
SQL> declare
2 cursor c_tool_list is
3 select descr
4 from hardware
5 where aisle = 1
6 and item between 1 and 500;
7
8 l_descr hardware.descr%type;
9 begin
10 open c_tool_list;
11 loop
12 fetch c_tool_list into l_descr;
13 exit when c_tool_list%notfound;
14 end loop;
15 close c_tool_list;
16 end;
bulk
SQL> declare
2 cursor c_tool_list is
3 select descr
4 from hardware
5 where aisle = 1
6 and item between 1 and 500;
7
8 type t_descr_list is
9 table of c_tool_list%rowtype;
10 l_descr_list t_descr_list;
11
12 begin
13 open c_tool_list;
14 fetch c_tool_list
15 bulk collect into l_descr_list;
16 close c_tool_list;
17 end;
71. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.
implicit cursor
71
SQL> begin
2 for i in (
3 select descr
4 from hardware
5 where aisle = 1
6 and item between 1 and 500 )
7 loop
8 [processing]
9 end loop;
10 end;
bulk
SQL> begin
2 for i in (
3 select descr
4 from hardware
5 where aisle = 1
6 and item between 1 and 500 )
7 loop
8 [processing]
9 end loop;
10 end;
SQL> select value
2 from v$parameter
3 where name = 'plsql_optimize_level';
VALUE
-----
2
72. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.
"But why bother?"
bulk1.sql
74. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.
it's more than just elapsed time
75. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.
SQL> exec dbms_monitor.session_trace_enable;
[repeat the demo]
SQL> exec dbms_monitor.session_trace_disable;
SQL ID: frmas27snufjy Plan Hash: 652399433
SELECT DESCR
FROM HARDWARE
call count cpu elapsed disk query current rows
------- ------- -------- ---------- ---------- ---------- ---------- ----------
Parse 1 0.00 0.00 0 0 0 0
Execute 1 0.00 0.00 0 0 0 0
Fetch 1000001 4.56 3.70 0 1000011 0 1000000
------- ------- -------- ---------- ---------- ---------- ---------- ----------
total 1000003 4.56 3.70 0 1000011 0 1000000
SQL ID: frmas27snufjy Plan Hash: 652399433
SELECT DESCR
FROM HARDWARE
call count cpu elapsed disk query current rows
------- ------ -------- ---------- ---------- ---------- ---------- ----------
Parse 1 0.00 0.00 0 0 0 0
Execute 1 0.00 0.00 0 0 0 0
Fetch 1 0.00 0.34 0 10100 0 1000000
------- ------ -------- ---------- ---------- ---------- ---------- ----------
total 3 0.00 0.34 0 10100 0 1000000
78. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.
all that data goes somewhere
78
13 open c_tool_list;
14 fetch c_tool_list
15 bulk collect into l_descr_list; -- 1,000,000 rows into an array
16 close c_tool_list;
79. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.
SQL> select * from v$mystats
2 where name like '%pga%'
3 /
NAME VALUE
------------------------------ ----------
session pga memory 201534672
session pga memory max 201534672
80. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.
200m of memory !
per session
81. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.
law of diminishing returns
82. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.
this was about pinning
82
83. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.
typical block is 8KB
83
84. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.
NAME VALUE
------------------------------ ----------
session pga memory 2127112
session pga memory max 2155912
SQL> declare
2 cursor c_tool_list is
3 select descr
4 from hardware;
5
6 type t_descr_list is table of c_tool_list%rowtype;
7 l_descr_list t_descr_list;
8
9 begin
10 open c_tool_list;
11 loop
12 fetch c_tool_list
13 bulk collect
14 into l_descr_list limit 500;
15 exit when c_tool_list%notfound;
16 end loop;
17 close c_tool_list;
18 end;
19 /
PL/SQL procedure successfully completed.
Elapsed: 00:00:00.10
85. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.
important !
85
86. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.
SQL> declare
2 cursor c_tool_list is
3 select descr
4 from hardware;
5
6 type t_descr_list is table of c_tool_list%rowtype;
7 l_descr_list t_descr_list;
8
9 begin
10 open c_tool_list;
11 loop
12 fetch c_tool_list
13 bulk collect
14 into l_descr_list limit 500;
1) array is replaced
15 exit when c_tool_list%notfound;
16 end loop;
17 close c_tool_list;
18 end;
19 /
2) < 500 = "not found"
[processing here!]
for i in 1 .. l_descr_list.count
87. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.
even easier ...
87
88. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.
implicit cursor
88
SQL> begin
2 for i in (
3 select descr
4 from hardware
5 where aisle = 1
6 and item between 1 and 500 )
7 loop
8 [processing]
9 end loop;
10 end;
bulk
SQL> begin
2 for i in (
3 select descr
4 from hardware
5 where aisle = 1
6 and item between 1 and 500 )
7 loop
8 [processing]
9 end loop;
10 end;
89. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.
guidelines
89
90. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.
1) single row
no change
91. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.
91
SQL> declare
2 l_descr hardware.descr%type;
3 begin
4 select descr
5 into l_descr
6 from hardware
7 where aisle = 1 -- single
8 and item = 1; -- row
9 end;
92. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.
2) finite and "reasonable"
93. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.
93
SQL> declare
2 type t_descr_list is
3 table of hardware.descr%type;
4 l_descr_list t_descr_list;
5 begin
6 select descr
7 bulk collect
8 into l_descr_list
9 from hardware
10 where aisle = 1
11 and item between 1 and 100;
12 end;
94. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.
note: subtle differences
94
95. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.
SQL> declare
2 l_descr hardware.descr%type;
3 begin
4 select descr
5 into l_descr
6 from hardware
7 where aisle = 0
8 and item = 0;
9 end;
10 /
declare
*
ERROR at line 1:
ORA-01403: no data found
ORA-06512: at line 4
SQL> declare
2 l_descr hardware.descr%type;
3 begin
4 select descr
5 into l_descr
6 from hardware
7 where aisle = 0
8 and item = 0;
9 dbms_output.put_line('YES!');
10 exception
11 when no_data_found then
12 dbms_output.put_line('NO!');
13 end;
14 /
NO!
96. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.
SQL> declare
2 type t_descr_list is table of hardware.descr%type;
3 l_descr_list t_descr_list;
4 begin
5 select descr
6 bulk collect
7 into l_descr_list
8 from hardware
9 where aisle = 0
10 and item = 0;
11 dbms_output.put_line('YES!');
12 exception
13 when no_data_found then
14 dbms_output.put_line('NO!');
15 end;
16 /
YES!
no exception
if l_descr_list.count = 0 then raise no_data_found; end if;
97. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.
3) unknown number of rows
97
98. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.
98
SQL> begin
2 for i in (
3 select descr
4 from hardware
5 where aisle = 1
6 and item between 1 and 500 )
7 loop
8 [processing]
9 end loop;
10 end;
99. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.
special case #1
99
100. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.
100
SQL> begin
2 for i in (
3 select descr
4 from hardware
5 where aisle = 1
6 and item between 1 and 500 )
7 loop
8
...
...
exit [early] when [some condition];
...
..
19 end loop;
20 end;
101. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.
special case #2
101
102. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.
dont forget this bit
103. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.
reduce network roundtrips
103
larger array sizes
104. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.
sidebar: SQL as well
104
105. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.
a little known feature
106. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.
SQL Net de-duplicate
107. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.
SQL> set autotrace traceonly stat
SQL> set arraysize 100
SQL> select aisle, item, descr
2 from hardware
3 where ...;
89405 rows selected.
Statistics
---------------------------------------------
4 recursive calls
0 db block gets
2452 consistent gets
0 physical reads
0 redo size
1198478 bytes sent via SQL*Net to client
107
108. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.
SQL> select aisle, item, descr
2 from hardware
3 where ...
4 order by 1,2;
89405 rows selected.
Statistics
--------------------------------------------
14 recursive calls
0 db block gets
1507 consistent gets
0 physical reads
0 redo size
618820 bytes sent via SQL*Net to client
109. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.
focus on blocks
109
110. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.
focus on roundtrips
110
111. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.
Part 2
writing data
111
112. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.
the same rules apply
112
113. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.
conventional DML in PLSQL
113
114. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.
114
SQL> declare
2 l_row hardware%rowtype;
3 begin
4 for i in 1 .. 100 loop
5 l_row.aisle := 1;
6 l_row.item := i;
7 insert into hardware values
8 ( l_row.aisle, l_row.item, l_row.descr) ;
9 end loop;
10 end;
11 /
PL/SQL procedure successfully completed.
115. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.
or even nicer (PLSQL only)
115
116. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.
116
SQL> declare
2 l_row hardware%rowtype;
3 begin
4 for i in 1 .. 100 loop
5 l_row.aisle := 1;
6 l_row.item := i;
7 insert into hardware values l_row;
8 end loop;
9 end;
10 /
PL/SQL procedure successfully completed.
117. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.
DML bulk = "bulk bind / FORALL"
117
118. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.
118
SQL> declare
2 type t_row_list is table of hardware%rowtype;
3 l_row t_row_list := t_row_list();
4 begin
5 for i in 1 .. 100 loop
6 l_row.extend;
7 l_row(i).aisle := 1;
8 l_row(i).item := i;
9 end loop;
10
11 forall i in 1 .. 100
12 insert into hardware values l_row(i);
13 end;
14 /
PL/SQL procedure successfully completed.
119. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.
"Is it worth the extra code?"
bulk2.sql
120. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.
it's better than that
bulk3.sql
122. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.
... the same rules apply
123. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.
all the data is somewhere
PGA memory
124. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.
law of diminishing returns
bulk4.sql
125. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.
no auto option :-(
126. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.
100-1000 typically fine
127. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.
one special case
128. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.
SQL> insert /*+ APPEND_VALUES */
2 into HARDWARE ( item ) values (1);
1 row created.
SQL> select item from HARDWARE;
select item from HARDWARE
*
ERROR at line 1:
ORA-12838: cannot read/modify an object after modifying it in parallel
SQL> commit;
Commit complete.
SQL> select item from HARDWARE;
ITEM
----------
1
129. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.
direct load
130. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.
high speed, massive data loads
bulk7.sql
131. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.
challenges with multi-row DML
132. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.
SQL> alter table hardware
2 add constraint
3 hardware_chk check ( item > 0 );
SQL> begin
2 insert into hardware ( item ) values (1);
3 insert into hardware ( item ) values (-1);
4 insert into hardware ( item ) values (2);
5 insert into hardware ( item ) values (3);
6 insert into hardware ( item ) values (4);
7 insert into hardware ( item ) values (-2);
8 end;
9 /
ERROR at line 1:
ORA-02290: check constraint (MCDONAC.HARDWARE_CHK) violated
ORA-06512: at line 3
SQL> select * from hardware;
no rows selected
133. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.
SQL> declare
2 type t_list is table of hardware.item%type;
3 l_rows t_list := t_list(1,-1,2,3,4,-2);
4 begin
5 forall i in 1 .. l_rows.count
6 insert into hardware ( item ) values (l_rows(i));
7 end;
8 /
declare
*
ERROR at line 1:
ORA-02290: check constraint (MCDONAC.HARDWARE_CHK) violated
ORA-06512: at line 5
134. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.
issue #1: which row cause the problem
135. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.
issue #2: none of the rows were stored
136. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.
SAVE EXCEPTIONS clause
bulk5.sql
137. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.
also consider error logging
bulk6.sql
138. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.
issue #3: sparseness
139. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.
SQL> declare
2 type t_num_list is table of hardware.item%type index by pls_integer;
3
4 val t_num_list;
5 begin
6
7 val(10) := 10;
8 val(11) := 20;
9 val(12) := 20;
10
11 forall i in 1 .. val.count
12 insert into hardware ( item ) values (val(i));
13
14 end;
15 /
declare
*
ERROR at line 1:
ORA-22160: element at index [1] does not exist
ORA-06512: at line 11
140. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.
SQL> declare
2 type t_num_list is table of hardware.item%type index by pls_integer;
3
4 val t_num_list;
5 begin
6
7 val(10) := 10;
8 val(11) := 20;
9 val(12) := 20;
10
11 forall i in val.first .. val.last
12 insert into hardware ( item ) values (val(i));
13
14 end;
15 /
PL/SQL procedure successfully completed.
141. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.
but ... contiguous is assumed
141
142. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.
SQL> declare
2 type t_num_list is table of hardware.item%type index by pls_integer;
3
4 val t_num_list;
5 begin
6
7 val(10) := 10;
8 -- val(11) := 20;
9 val(12) := 20;
10
11 forall i in val.first .. val.last
12 insert into hardware ( item ) values (val(i));
13
14 end;
15 /
declare
*
ERROR at line 1:
ORA-22160: element at index [11] does not exist
ORA-06512: at line 11
143. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.
INDICES OF clause
143
144. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.
SQL> declare
2 type t_num_list is table of hardware.item%type index by pls_integer;
3
4 val t_num_list;
5 begin
6
7 val(10) := 10;
8 -- val(11) := 20;
9 val(12) := 20;
10
11 forall i in indices of val
12 insert into hardware ( item ) values (val(i));
13
14 end;
15 /
PL/SQL procedure successfully completed.
145. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.
homework :-)
145
146. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.
VALUES OF clause
146
147. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.
"How to get array benefits with SQL?"
148. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.
SQL does this automatically...
149. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.
full table scans
index range scans
prefetch
150. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.
use SQL in preference to PLSQL where possible
150
for another day
151. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.
wrap up
151
152. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.
“relational”
not
“rowlational”
153. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.
process data in sets not rows
154. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.
use bulk collect / bulk bind where appropriate