Python generators for efficient data streaming. Presented at Amazon COMMIT 2015

1. Maxym Kharchenko & m@ team
Writing efficient Python code with
pipelines and generators

2. Agenda

3. Python is all about streaming (a.k.a. iteration)

4. Streaming in Python
# Lists
db_list = ['db1', 'db2', 'db3']
for db in db_list:
print db
# Dictionaries
host_cpu = {'avg': 2.34, 'p99': 98.78, 'min': 0.01}
for stat in host_cpu:
print "%s = %s" % (stat, host_cpu[stat])
# Files, strings
file = open("/etc/oratab")
for line in file:
for word in line.split(" "):
print word
# Whatever is coming out of get_things()
for thing in get_things():
print thing

5. Quick example: Reading records from a file
def print_databases():
""" Read /etc/oratab and print database names """
file = open("/etc/oratab", 'r')
while True:
line = file.readline() # Get next line
# Check for empty lines
if len(line) == 0 and not line.endswith('n'):
break
# Parsing oratab line into components
db_line = line.strip()
db_info_array = db_line.split(':')
db_name = db_info_array[0]
print db_name
file.close()

6. Reading records from a file: with “streaming”
def print_databases():
""" Read /etc/oratab and print database names """
with open("/etc/oratab") as file:
for line in file:
print line.strip().split(':')[0]

7. Style matters!

8. Ok, let’s do something useful with streaming
 We have a bunch of ORACLE listener logs
 Let’s parse them for “client IPs”
21-AUG-2015 21:29:56 *
(CONNECT_DATA=(SID=orcl)(CID=(PROGRAM=)(HOST=_
_jdbc__)(USER=))) *
(ADDRESS=(PROTOCOL=tcp)(HOST=10.107.137.91)(PO
RT=43105)) * establish * orcl * 0
 And find where the clients are coming from

9. First attempt at listener log parser
def parse_listener_log(log_name):
""" Parse listener log and return clients
"""
client_hosts = []
with open(log_name) as listener_log:
for line in listener_log:
host_match = <regex magic>
if host_match:
host = <regex magic>
client_hosts.append(host)
return client_hosts

10. First attempt at listener log parser
def parse_listener_log(log_name):
""" Parse listener log and return clients
"""
client_hosts = []
with open(log_name) as listener_log:
for line in listener_log:
host_match = <regex magic>
if host_match:
host = <regex magic>
client_hosts.append(host)
return client_hosts
MEMORY
WASTE!
Stores all
results until
return
BLOCKING!
Does NOT
return until
the entire log
is processed

11. Generators for efficiency
def parse_listener_log(log_name):
""" Parse listener log and return clients
"""
client_hosts = []
with open(log_name) as listener_log:
for line in listener_log:
host_match = <regex magic>
if host_match:
host = <regex magic>
client_hosts.append(host)
return client_hosts

12. Generators for efficiency
def parse_listener_log(log_name):
""" Parse listener log and return clients
"""
client_hosts = []
with open(log_name) as listener_log:
for line in listener_log:
host_match = <regex magic>
if host_match:
host = <regex magic>
client_hosts.append(host)
return client_hosts

13. Generators for efficiency
def parse_listener_log(log_name):
""" Parse listener log and return clients
"""
with open(log_name) as listener_log:
for line in listener_log:
host_match = <regex magic>
if host_match:
host = <regex magic>
yield hostAdd this !

14. Generators in a nutshell
def test_generator():
""" Test generator """
print "ENTER()"
for i in range(5):
print "yield i=%d" % i
yield i
print "EXIT()"
# MAIN
for i in test_generator():
print "RET=%d" % i
ENTER()
yield i=0
RET=0
yield i=1
RET=1
yield i=2
RET=2
yield i=3
RET=3
yield i=4
RET=4
EXIT()

15. Nongenerators in a nutshell
def test_nongenerator():
""" Test no generator """
result = []
print "ENTER()"
for i in range(5):
print "add i=%d" % i
result.append(i)
print "EXIT()"
return result
# MAIN
for i in test_nongenerator():
print "RET=%d" % i
ENTER()
add i=0
add i=1
add i=2
add i=3
add i=4
EXIT()
RET=0
RET=1
RET=2
RET=3
RET=4

16. Generators to Pipelines
Generator
(extractor)
1 second
per record
100,000
1st:
1 second
100,000
Generator
(filter: 1/2)
2 seconds
per record
Generator
(mapper)
5 seconds
per record
50,000
1st:
5 seconds
50,000
1st:
10 seconds

17. Generator pipelining in Python
file_handles = open_files(LISTENER_LOGS)
log_lines = extract_lines(file_handles)
client_hosts = extract_client_ips(log_lines)
for host in client_hosts:
print host
Open
files
Extract
lines
Extract
IPs
File
names
File
handles
File
lines
Client
IPs

18. Generators for simplicity
def open_files(file_names):
""" GENERATOR: file name -> file handle """
for file in file_names:
yield open(file)

19. Generators for simplicity
def extract_lines(file_handles):
""" GENERATOR: File handles -> file lines
Similar to UNIX: cat file1, file2, …
"""
for file in file_handles:
for line in file:
yield line

20. Generators for simplicity
def extract_client_ips(lines):
""" GENERATOR: Extract client host
"""
host_regex = re.compile('(HOST=(S+))(PORT=')
for line in lines:
line_match = host_regex.search(line)
if line_match:
yield line_match.groups(0)[0]

21. Developer’s bliss:
simple input, simple output, trivial function body

22. Then, pipeline the results

23. But, really …
Open
files
Extract
lines
IP ->
host
name
File
names
File
handles
File
lines
Client
hosts
Locate
files
Filter
db=orcl
Filter
proto=
TCP
db=orcl
lines
db=orcl
lines
db=orcl
&
prot=TCP
Extract
clients
Client
IPs
Client
hosts
Db
writer
Client
hosts
Text
writer

24. Why generators ?
 Simple functions that are easy to write and understand
 Non blocking operations:
 TOTAL execution time: faster
 FIRST RESULTS: much faster
 Efficient use of memory
 Potential for parallelization and ASYNC processing

25. Special thanks to David Beazley …
 For this: http://www.dabeaz.com/generators-uk/GeneratorsUK.pdf

26. Thank you!