Chose a better scripting language. How to manage ORACLE databases with Python.

1. Maxym Kharchenko
Manage ORACLE databases
with Python

2. Whoami
■ Started as a database kernel developer with C and C++
■ ORACLE DBA for ~ 14 years: + SQL, PL/SQL and Perl
■ Persistence Engineer for the last 4: + Python, R and Scala
■ OCM, ORACLE Ace Associate
■ Blog: http://intermediatesql.com
■ Twitter: @maxymkh

3. Agenda
■ Talk about why Python is awesome
■ Design ORACLE multi db “ping” in Python
■ (hopefully) Demo it

4. The job of an engineer is
to make complex things simple

5. Why script outside the database ?

6. Do I have a good datafilebackup ?
SELECT …
FROM v$backup_datafile
 v$backup_datafile
 Backup files actually exist
 and can be validated
 Backup size is plausible
 On the end storage
 Network transfer as well
 No critical errors in logs
 Etc

7. Some things are
just difficult to do in a database

8. You will, generally, get better Zen
i := i + 1 i += 1

9. You will, generally, get better Zen

10. You will, generally, get better Zen

11. You, probably, have >1 database
ORACLE ORACLE
ORACLE ORACLE ORACLE
ORACLE ORACLE ORACLE
ORACLE MySql MySql
Postgres Cassandra
S3 S3

12. So, why Python ?

13. There should be one way to do it
And it should be obvious

14. Python enforces good coding practices

15. Python enforces good coding practices
foreach my $p (keys %$p1) {if(exists($p1->{$p})
and exists($p2->{$p})) {
if(uc($p1->{$p}) eq uc($p2->{$p})) {$diff-
>{SAME}->{$p} = 1;
} else {$diff->{DIFF}->{$p} = 1;
}}} elsif(! exists($p2->{$p})) {$diff-
>{ONLY_IN_1}->{$p} = 1;}}

16. Python enforces good coding practices
for p in db1_params:
if p in db2_params:
if db1_params[p] == db2_params[p]:
same_parameters.append(p)
else:
diff_parameters.append(p)
else:
only_in_1.append(p)

17. Python should come
with all the batteries included

18. Python interfaces to everything
import urllib
import json
url =
"http://api.openweathermap.org/data/2.5/weather?q=Seattle
,WA"
url_obj = urllib.urlopen(url)
data = url_obj.read().strip()
data = json.loads(data)
print(data['weather'])

19. Python interfaces to everything
import cx_Oracle
sql = "SELECT user FROM dual"
conn = cx_Oracle.connect('scott/tiger@orcl')
cursor = conn.cursor()
cursor.execute(sql)
db_user = cursor.fetchall()

20. Python interfaces to everything
import boto
from boto.s3.key import Key
s3 = boto.connect_s3()
bucket = s3.get_bucket('my_cats')
k.key = 'lazy_cat.jpg'
k.set_contents_from_filename('/tmp/lazy_cat.jpg')

21. Python is pretty popular

22. Python is pretty popular

23. If you know any scripting language
you (almost) know Python

24. Python is similar to other languages
def is_accessible(db_name):
""" Check if database is accessible """
ret = False
db_status = ping_db(db_name)
if "ACTIVE" == db_status:
ret = True
return ret

25. But: Spaces are first class citizens
def print_databases():
""" Print all databases from /etc/oratab """
with open("/etc/oratab") as file:
for line in file:
if line:
print line.strip().split(':')[0]

26. But: Spaces are first class citizens
def print_databases():
""" Print all databases from /etc/oratab ""“
with open("/etc/oratab") as file:
for line in file:
if line:
print line.strip().split(':')[0]
File "./a.py", line 7
print line.strip().split(':')[0]
^
IndentationError:
expected an indented block

27. Compare to Perl
sub print_databases() {
open(my $f, "<", "/etc/oratab")
or die ("Unable to open: $!");
while(my $line = <$f>) {
if ($line =~ /S+/) {
my @aItems = split(':', $line);
print $aItems[0] . "n";
}
}
close($f);
}

28. Compare to Perl
sub print_databases(){open(my $f, "<", "/etc/oratab") or
die ("Unable to open: $!"); while(my $line = <$f>) {
if ($line =~ /S+/) {my @aItems = split(':', $line);
print $aItems[0] . "n";}} close($f);}

29. Functions are first class citizens too

30. Fun with functions
def outer_function(parameter_function):
""" This function accepts function as a parameter """
def inner_function(inner_parameter):
""" This is a nested function """
return inner_parameter
# This returns a function from a function
return inner_function(parameter_function)
# This "calls" function return value as a function
print outer_function(external_function)()

31. Fun with functions: Decorators
def do_stuff():
result = heavy_processing()
def do_stuff():
start = time()
result = heavy_processing()
end = time()
print "Elapsed: %f" % (end-start)
def do_stuff2():
…
def do_stuff3076():

32. Fun with functions: Decorators
def timeit(func):
""" Generic time profiling function """
def time_exec(*args, **kwargs):
start_time = time()
ret = func(*args, **kwargs)
end = time()
print "Elapsed: %f" % (end-start)
return ret
return time_exec

33. Fun with functions: Decorators
do_stuff = timeit(do_stuff)
@timeit
def do_stuff():
…
@timeit
def do_stuff2():
…
@timeit
def do_stuff3076():
…

34. Learn to think Pythonian. It helps!
def print_databases():
file = open('/etc/oratab', 'r')
while True:
line = file.readline()
if len(line) == 0 and not line.endswith('n'):
break
print line.strip().split(':')[0]
file.close()
def print_databases():
with open('/etc/oratab') as file:
for line in file:
print line.strip().split(':')[0]

35. Python and ORACLE

36. Database “multi ping” tool
■ Ping ORACLE database
▪ Report AVAILABLE / NOT AVAILABLE status
■ Option to ping multiple databases
▪ Preferably in parallel
■ Bonus: track execution timing

37. Demo

38. cx_Oracle: Running SQL
import cx_Oracle
def is_db_alive(db_name):
is_alive = False
try:
conn = cx_Oracle.connect("user/password@%s" % db_name)
cursor = conn.cursor()
cursor.execute("SELECT user FROM dual")
except:
is_alive = False
else:
is_alive = True
return is_alive

39. Database ping
> dbping.py c15lv1
PING [c15lv1]: OK
> dbping.py c15lv2
PING [c15lv2]: UNAVAILABLE

40. Database “multi ping”
import dbping
def multi_dbping(db_list, ping_routine):
""" Ping all databases in a list """
for db in db_list:
ping_routine(db)
> dbping_list.py c15lv1 c15lv2 c15lv3
PING [c15lv1]: OK
PING [c15lv2]: UNAVAILABLE
PING [c15lv3]: OK

41. Parallel database “multi ping”
import multiprocessing
def parallel_ping(db_list, target=dbping.print_dbping):
""" Ping db_list databases in parallel """
jobs = []
for d in db_list:
p = multiprocessing.Process(
target=target, args=(d,)
)
jobs.append(p)
p.start()
for p in jobs:
p.join()

42. Parallel database “multi ping”
> dbping_parallel.py c15lv1 c15lv2 c15lv3 c15lv4
PING [c15lv1]: OK
PING [c15lv3]: OK
PING [c15lv4]: OK
PING [c15lv2]: UNAVAILABLE

43. Decorator: Execution Timing
def timeit(func):
""" Generic time profiling function """
def time_exec(*args, **kwargs):
start_time = time()
ret = func(*args, **kwargs)
ela = time() - start_time
print “tElapsed: %.3f seconds" % ela
return ret
return time_exec

44. Execution Timing:
> dbping_parallel_timing.py c15lv1 c15lv2 c15lv3
PING [c15lv3]: OK
Elapsed: 1.186 seconds
PING [c15lv1]: OK
Elapsed: 2.309 seconds
PING [c15lv2]: UNAVAILABLE
Elapsed: 22.511 seconds
@timeit
def print_dbping_with_timing(db_name):
return dbping.print_dbping(db_name)

45. How to start with Python
■ Lots of free resources on the web
▪ Tutorials
▪ Documentation
▪ Stackoverflow.com
▪ “Play” environments
▪ Even books
■ Python self documentation:
▪ dir()
▪ help()

46. Thank you!