SlideShare uma empresa Scribd logo
1 de 91
Baixar para ler offline
Hidden Treasures
      of the Standard Library
      Doug Hellmann
      PyCon
      February, 2011




Sunday, March 20, 2011
Python Module of the Week:
      “I read the docs, so you don’t have to.”




Sunday, March 20, 2011
Read This!
      Pre-order now




Sunday, March 20, 2011
Goals   Usable Tips


Sunday, March 20, 2011
Look Around   Change Focus


Sunday, March 20, 2011
Look Around   100+ Modules


Sunday, March 20, 2011
Full Name    User Name                Email



           Doug Hellmann       dhellmann   doug.hellmann@gmail.com


            Guest account,
                                 guest       guest@example.com
               no login




                 Parsing Structured Data    csv dialects


Sunday, March 20, 2011
Test Database
             1     #!/bin/sh
             2
             3     rm -f test.db
             4
             5     sqlite3 test.db <<EOF
             6
             7     CREATE TABLE users (
             8        fullname text,
             9        username text,
            10        email    text
            11     );
            12
            13     INSERT INTO users (fullname, username, email)
            14     VALUES ('Doug Hellmann', 'dhellmann', 'doug.hellmann@gmail.com');
            15
            16     INSERT INTO users (fullname, username, email)
            17     VALUES ('Guest account, no login', 'guest', 'guest@example.com');
            18
            19     EOF
            20



Sunday, March 20, 2011
Test Database
             1     #!/bin/sh
             2
             3     rm -f test.db
             4
             5     sqlite3 test.db <<EOF
             6
             7     CREATE TABLE users (
             8        fullname text,
             9        username text,
            10        email    text
            11     );
            12
            13     INSERT INTO users (fullname, username, email)
            14     VALUES ('Doug Hellmann', 'dhellmann', 'doug.hellmann@gmail.com');
            15
            16     INSERT INTO users (fullname, username, email)
            17     VALUES ('Guest account, no login', 'guest', 'guest@example.com');
            18
            19     EOF
            20



Sunday, March 20, 2011
Database Contents
       $ ./hidden_stdlib/csv_mkdb.sh


       $ sqlite3 -noheader test.db 'select * from users'
       Doug Hellmann|dhellmann|doug.hellmann@gmail.com
       Guest account, no login|guest|guest@example.com




Sunday, March 20, 2011
Database Contents
       $ ./hidden_stdlib/csv_mkdb.sh


       $ sqlite3 -noheader test.db 'select * from users'
       Doug Hellmann|dhellmann|doug.hellmann@gmail.com
       Guest account, no login|guest|guest@example.com




Sunday, March 20, 2011
csv Dialect
              9 csv.register_dialect('pipes', delimiter='|')
             10
             11 reader = csv.reader(sys.stdin, dialect='pipes')
             12 for row in reader:
             13     print row




Sunday, March 20, 2011
csv Dialect
              9 csv.register_dialect('pipes', delimiter='|')
             10
             11 reader = csv.reader(sys.stdin, dialect='pipes')
             12 for row in reader:
             13     print row




Sunday, March 20, 2011
csv Dialect
              9 csv.register_dialect('pipes', delimiter='|')
             10
             11 reader = csv.reader(sys.stdin, dialect='pipes')
             12 for row in reader:
             13     print row




Sunday, March 20, 2011
Database Contents
       $ ./hidden_stdlib/csv_mkdb.sh


       $ sqlite3 -noheader test.db 'select * from users'
       Doug Hellmann|dhellmann|doug.hellmann@gmail.com
       Guest account, no login|guest|guest@example.com


       $ sqlite3 -noheader test.db 'select * from users' 
          | python ./hidden_stdlib/csv_pipes.py
       ['Doug Hellmann', 'dhellmann', 'doug.hellmann@gmail.com']
       ['Guest account, no login', 'guest', 'guest@example.com']




Sunday, March 20, 2011
Database Contents
       $ ./hidden_stdlib/csv_mkdb.sh


       $ sqlite3 -noheader test.db 'select * from users'
       Doug Hellmann|dhellmann|doug.hellmann@gmail.com
       Guest account, no login|guest|guest@example.com


       $ sqlite3 -noheader test.db 'select * from users' 
          | python ./hidden_stdlib/csv_pipes.py
       ['Doug Hellmann', 'dhellmann', 'doug.hellmann@gmail.com']
       ['Guest account, no login', 'guest', 'guest@example.com']




Sunday, March 20, 2011
Custom Column Types   sqlite3


Sunday, March 20, 2011
Custom SQLite Column Types
             42          # Insert the objects into the database
             43          print 'Inserting:'
             44          to_save = [ (MyObj('this is a value to save'),),
             45                       (MyObj(42),),
             46                       ]
             47          cursor.executemany(
             48              "insert into obj (data) values (?)",
             49              to_save)
             50
             51          # Query the database for the objects just saved
             52          print 'nQuerying:'
             53          cursor.execute("select id, data from obj")
             54          for obj_id, obj in cursor.fetchall():
             55              print 'Retrieved', obj_id, type(obj), obj




Sunday, March 20, 2011
Custom SQLite Column Types
             42          # Insert the objects into the database
             43          print 'Inserting:'
             44          to_save = [ (MyObj('this is a value to save'),),
             45                       (MyObj(42),),
             46                       ]
             47          cursor.executemany(
             48              "insert into obj (data) values (?)",
             49              to_save)
             50
             51          # Query the database for the objects just saved
             52          print 'nQuerying:'
             53          cursor.execute("select id, data from obj")
             54          for obj_id, obj in cursor.fetchall():
             55              print 'Retrieved', obj_id, type(obj), obj




Sunday, March 20, 2011
Custom SQLite Column Types
             42          # Insert the objects into the database
             43          print 'Inserting:'
             44          to_save = [ (MyObj('this is a value to save'),),
             45                       (MyObj(42),),
             46                       ]
             47          cursor.executemany(
             48              "insert into obj (data) values (?)",
             49              to_save)
             50
             51          # Query the database for the objects just saved
             52          print 'nQuerying:'
             53          cursor.execute("select id, data from obj")
             54          for obj_id, obj in cursor.fetchall():
             55              print 'Retrieved', obj_id, type(obj), obj




Sunday, March 20, 2011
Custom SQLite Column Types
              9 class MyObj(object):
             10     def __init__(self, arg):
             11         self.arg = arg
             12     def __str__(self):
             13         return 'MyObj(%r)' % self.arg




Sunday, March 20, 2011
Custom SQLite Column Types
             17     def adapter_func(obj):
             18         """memory -> storage"""
             19         print 'Saving:', obj
             20         return pickle.dumps(obj)
             21
             22     sqlite3.register_adapter(MyObj, adapter_func)
             23
             24     def converter_func(data):
             25         """storage -> memory"""
             26         return pickle.loads(data)
             27
             28     sqlite3.register_converter("MyObj", converter_func)




Sunday, March 20, 2011
Custom SQLite Column Types
             17     def adapter_func(obj):
             18         """memory -> storage"""
             19         print 'Saving:', obj
             20         return pickle.dumps(obj)
             21
             22     sqlite3.register_adapter(MyObj, adapter_func)
             23
             24     def converter_func(data):
             25         """storage -> memory"""
             26         return pickle.loads(data)
             27
             28     sqlite3.register_converter("MyObj", converter_func)




Sunday, March 20, 2011
Custom SQLite Column Types
             17     def adapter_func(obj):
             18         """memory -> storage"""
             19         print 'Saving:', obj
             20         return pickle.dumps(obj)
             21
             22     sqlite3.register_adapter(MyObj, adapter_func)
             23
             24     def converter_func(data):
             25         """storage -> memory"""
             26         return pickle.loads(data)
             27
             28     sqlite3.register_converter("MyObj", converter_func)




Sunday, March 20, 2011
Custom SQLite Column Types
             17     def adapter_func(obj):
             18         """memory -> storage"""
             19         print 'Saving:', obj
             20         return pickle.dumps(obj)
             21
             22     sqlite3.register_adapter(MyObj, adapter_func)
             23
             24     def converter_func(data):
             25         """storage -> memory"""
             26         return pickle.loads(data)
             27
             28     sqlite3.register_converter("MyObj", converter_func)




Sunday, March 20, 2011
Custom SQLite Column Types
             30 with sqlite3.connect(
             31          'type_demo.db',
             32          detect_types=sqlite3.PARSE_DECLTYPES) as conn:
             33     # Create a table with column of type "MyObj"
             34     conn.execute("""
             35     create table if not exists obj (
             36          id    integer primary key autoincrement not null,
             37          data MyObj
             38     )
             39     """)




Sunday, March 20, 2011
Custom SQLite Column Types
             30 with sqlite3.connect(
             31          'type_demo.db',
             32          detect_types=sqlite3.PARSE_DECLTYPES) as conn:
             33     # Create a table with column of type "MyObj"
             34     conn.execute("""
             35     create table if not exists obj (
             36          id    integer primary key autoincrement not null,
             37          data MyObj
             38     )
             39     """)




Sunday, March 20, 2011
Custom SQLite Column Types
       $ python hidden_stdlib/sqlite3_custom_type.py
       Inserting:
       Saving: MyObj('this is a value to save')
       Saving: MyObj(42)


       Querying:
       Retrieved 1 <class '__main__.MyObj'> MyObj('this is a value to
       save')
       Retrieved 2 <class '__main__.MyObj'> MyObj(42)




Sunday, March 20, 2011
Custom SQLite Column Types
       $ python hidden_stdlib/sqlite3_custom_type.py
       Inserting:
       Saving: MyObj('this is a value to save')
       Saving: MyObj(42)


       Querying:
       Retrieved 1 <class '__main__.MyObj'> MyObj('this is a value to
       save')
       Retrieved 2 <class '__main__.MyObj'> MyObj(42)




Sunday, March 20, 2011
Custom SQLite Column Types
       $ python hidden_stdlib/sqlite3_custom_type.py
       Inserting:
       Saving: MyObj('this is a value to save')
       Saving: MyObj(42)


       Querying:
       Retrieved 1 <class '__main__.MyObj'> MyObj('this is a value to
       save')
       Retrieved 2 <class '__main__.MyObj'> MyObj(42)




Sunday, March 20, 2011
Signed Serialized Data   hmac


Sunday, March 20, 2011
hmac
             11     message = 'The message'
             12     encoded_message = pickle.dumps(message)
             13
             14     digest_maker = hmac.new('shared-secret-value')
             15     digest_maker.update(encoded_message)
             16     signature = digest_maker.hexdigest()
             17
             18     print 'Outgoing signature:', signature
             19
             20     # Simulate sending the message
             21     buffer = StringIO('%sn%dn%s' % (signature,
             22                                       len(encoded_message),
             23                                       encoded_message
             24                                       ))




Sunday, March 20, 2011
hmac
             11     message = 'The message'
             12     encoded_message = pickle.dumps(message)
             13
             14     digest_maker = hmac.new('shared-secret-value')
             15     digest_maker.update(encoded_message)
             16     signature = digest_maker.hexdigest()
             17
             18     print 'Outgoing signature:', signature
             19
             20     # Simulate sending the message
             21     buffer = StringIO('%sn%dn%s' % (signature,
             22                                       len(encoded_message),
             23                                       encoded_message
             24                                       ))




Sunday, March 20, 2011
hmac
             11     message = 'The message'
             12     encoded_message = pickle.dumps(message)
             13
             14     digest_maker = hmac.new('shared-secret-value')
             15     digest_maker.update(encoded_message)
             16     signature = digest_maker.hexdigest()
             17
             18     print 'Outgoing signature:', signature
             19
             20     # Simulate sending the message
             21     buffer = StringIO('%sn%dn%s' % (signature,
             22                                       len(encoded_message),
             23                                       encoded_message
             24                                       ))




Sunday, March 20, 2011
hmac
             26     # "Receive" the message
             27     read_signature = buffer.readline().rstrip()
             28     message_len = int(buffer.readline())
             29     read_message = buffer.read(message_len)
             30
             31     # Check the signature of the incoming data
             32     digest_maker = hmac.new('shared-secret-value',
             33                             read_message)
             34     computed_signature = digest_maker.hexdigest()
             35     print 'Computed signature:', signature




Sunday, March 20, 2011
hmac
             26     # "Receive" the message
             27     read_signature = buffer.readline().rstrip()
             28     message_len = int(buffer.readline())
             29     read_message = buffer.read(message_len)
             30
             31     # Check the signature of the incoming data
             32     digest_maker = hmac.new('shared-secret-value',
             33                             read_message)
             34     computed_signature = digest_maker.hexdigest()
             35     print 'Computed signature:', signature




Sunday, March 20, 2011
hmac
             37 if computed_signature == read_signature:
             38     print 'nValid message, processed'
             39     safe_message = pickle.loads(read_message)
             40     print 'Message:', safe_message
             41 else:
             42     raise ValueError('Invalid message, discarded')




Sunday, March 20, 2011
hmac
             37 if computed_signature == read_signature:
             38     print 'nValid message, processed'
             39     safe_message = pickle.loads(read_message)
             40     print 'Message:', safe_message
             41 else:
             42     raise ValueError('Invalid message, discarded')




Sunday, March 20, 2011
hmac
       $ python hidden_stdlib/hmac_message_signing.py
       Outgoing signature: 3498b6fa42a75dc4603da6f4c67505a2
       Computed signature: 3498b6fa42a75dc4603da6f4c67505a2


       Valid message, processed
       Message: The message




Sunday, March 20, 2011
Serializing Objects   json


Sunday, March 20, 2011
json
             10 def convert_to_builtin_type(obj):
             11     """object->dictionary"""
             12     print 'convert_to_builtin_type(%r)' % obj
             13     class_name = obj.__class__.__name__
             14     module_name = obj.__module__
             15     digest_maker = hmac.new('PyCon2011',
             16                              module_name + class_name)
             17     signature = digest_maker.hexdigest()
             18     d = { '__class__':class_name,
             19           '__module__':module_name,
             20           '__signature__':signature,
             21           }
             22     d.update(obj.__dict__)
             23     return d
             24
             25 obj = json_myobj.MyObj('instance value goes here')
             26 print json.dumps(obj, default=convert_to_builtin_type)




Sunday, March 20, 2011
json
             10 def convert_to_builtin_type(obj):
             11     """object->dictionary"""
             12     print 'convert_to_builtin_type(%r)' % obj
             13     class_name = obj.__class__.__name__
             14     module_name = obj.__module__
             15     digest_maker = hmac.new('PyCon2011',
             16                              module_name + class_name)
             17     signature = digest_maker.hexdigest()
             18     d = { '__class__':class_name,
             19           '__module__':module_name,
             20           '__signature__':signature,
             21           }
             22     d.update(obj.__dict__)
             23     return d
             24
             25 obj = json_myobj.MyObj('instance value goes here')
             26 print json.dumps(obj, default=convert_to_builtin_type)




Sunday, March 20, 2011
json
             10 def convert_to_builtin_type(obj):
             11     """object->dictionary"""
             12     print 'convert_to_builtin_type(%r)' % obj
             13     class_name = obj.__class__.__name__
             14     module_name = obj.__module__
             15     digest_maker = hmac.new('PyCon2011',
             16                              module_name + class_name)
             17     signature = digest_maker.hexdigest()
             18     d = { '__class__':class_name,
             19           '__module__':module_name,
             20           '__signature__':signature,
             21           }
             22     d.update(obj.__dict__)
             23     return d
             24
             25 obj = json_myobj.MyObj('instance value goes here')
             26 print json.dumps(obj, default=convert_to_builtin_type)




Sunday, March 20, 2011
json
             10 def convert_to_builtin_type(obj):
             11     """object->dictionary"""
             12     print 'convert_to_builtin_type(%r)' % obj
             13     class_name = obj.__class__.__name__
             14     module_name = obj.__module__
             15     digest_maker = hmac.new('PyCon2011',
             16                              module_name + class_name)
             17     signature = digest_maker.hexdigest()
             18     d = { '__class__':class_name,
             19           '__module__':module_name,
             20           '__signature__':signature,
             21           }
             22     d.update(obj.__dict__)
             23     return d
             24
             25 obj = json_myobj.MyObj('instance value goes here')
             26 print json.dumps(obj, default=convert_to_builtin_type)




Sunday, March 20, 2011
json
       $ python hidden_stdlib/json_dump_default.py
       convert_to_builtin_type(<MyObj(instance value goes here)>)
       {"s": "instance value goes here", "__module__": "json_myobj",
       "__signature__": "426f662f9fe3b3533d9ce7f9dcf8af77",
       "__class__": "MyObj"}




Sunday, March 20, 2011
json
       $ python hidden_stdlib/json_dump_default.py
       convert_to_builtin_type(<MyObj(instance value goes here)>)
       {"s": "instance value goes here", "__module__": "json_myobj",
       "__signature__": "426f662f9fe3b3533d9ce7f9dcf8af77",
       "__class__": "MyObj"}




Sunday, March 20, 2011
json
       $ python hidden_stdlib/json_dump_default.py
       convert_to_builtin_type(<MyObj(instance value goes here)>)
       {"s": "instance value goes here", "__module__": "json_myobj",
       "__signature__": "426f662f9fe3b3533d9ce7f9dcf8af77",
       "__class__": "MyObj"}




Sunday, March 20, 2011
json
       $ python hidden_stdlib/json_dump_default.py
       convert_to_builtin_type(<MyObj(instance value goes here)>)
       {"s": "instance value goes here", "__module__": "json_myobj",
       "__signature__": "426f662f9fe3b3533d9ce7f9dcf8af77",
       "__class__": "MyObj"}




Sunday, March 20, 2011
json
       $ python hidden_stdlib/json_dump_default.py
       convert_to_builtin_type(<MyObj(instance value goes here)>)
       {"s": "instance value goes here", "__module__": "json_myobj",
       "__signature__": "426f662f9fe3b3533d9ce7f9dcf8af77",
       "__class__": "MyObj"}




Sunday, March 20, 2011
json
              9 def dict_to_object(d):
             10     if '__class__' not in d:
             11         return d
             12
             13     class_name = d.pop('__class__')
             14     module_name = d.pop('__module__')
             15     signature = d.pop('__signature__')
             16
             17     digest_maker = hmac.new('PyCon2011',
             18                              module_name + class_name)
             19     expected_signature = digest_maker.hexdigest()
             20     if signature != expected_signature:
             21         raise ValueError('Invalid signature')




Sunday, March 20, 2011
json
              9 def dict_to_object(d):
             10     if '__class__' not in d:
             11         return d
             12
             13     class_name = d.pop('__class__')
             14     module_name = d.pop('__module__')
             15     signature = d.pop('__signature__')
             16
             17     digest_maker = hmac.new('PyCon2011',
             18                              module_name + class_name)
             19     expected_signature = digest_maker.hexdigest()
             20     if signature != expected_signature:
             21         raise ValueError('Invalid signature')




Sunday, March 20, 2011
json
              9 def dict_to_object(d):
             10     if '__class__' not in d:
             11         return d
             12
             13     class_name = d.pop('__class__')
             14     module_name = d.pop('__module__')
             15     signature = d.pop('__signature__')
             16
             17     digest_maker = hmac.new('PyCon2011',
             18                              module_name + class_name)
             19     expected_signature = digest_maker.hexdigest()
             20     if signature != expected_signature:
             21         raise ValueError('Invalid signature')




Sunday, March 20, 2011
json
             23          print 'Loading "%s" from "%s"' % 
             24              (class_name, module_name)
             25          module = __import__(module_name)
             26          class_ = getattr(module, class_name)
             27
             28          args = dict( (key.encode('ascii'), value)
             29                       for key, value in d.items())
             30          print 'Instantiating with', args
             31
             32          inst = class_(**args)
             33          return inst




Sunday, March 20, 2011
json
             23          print 'Loading "%s" from "%s"' % 
             24              (class_name, module_name)
             25          module = __import__(module_name)
             26          class_ = getattr(module, class_name)
             27
             28          args = dict( (key.encode('ascii'), value)
             29                       for key, value in d.items())
             30          print 'Instantiating with', args
             31
             32          inst = class_(**args)
             33          return inst




Sunday, March 20, 2011
json
             23          print 'Loading "%s" from "%s"' % 
             24              (class_name, module_name)
             25          module = __import__(module_name)
             26          class_ = getattr(module, class_name)
             27
             28          args = dict( (key.encode('ascii'), value)
             29                       for key, value in d.items())
             30          print 'Instantiating with', args
             31
             32          inst = class_(**args)
             33          return inst




Sunday, March 20, 2011
json
             23          print 'Loading "%s" from "%s"' % 
             24              (class_name, module_name)
             25          module = __import__(module_name)
             26          class_ = getattr(module, class_name)
             27
             28          args = dict( (key.encode('ascii'), value)
             29                       for key, value in d.items())
             30          print 'Instantiating with', args
             31
             32          inst = class_(**args)
             33          return inst




Sunday, March 20, 2011
json
             35 for encoded_object in [
             36     '''
             37     [{"s": "instance value goes here",
             38        "__signature__": "426f662f9fe3b3533d9ce7f9dcf8af77",
             39        "__module__": "json_myobj", "__class__": "MyObj"}]
             40     ''',
             41
             42     # careful!
             43     '''
             44     [{"path": "/etc/passwd",
             45        "__signature__": "426f662f9fe3b3533d9ce7f9dcf8af77",
             46        "__module__": "os", "__class__": "unlink"}]
             47     ''',
             48     ]:




Sunday, March 20, 2011
json
             35 for encoded_object in [
             36     '''
             37     [{"s": "instance value goes here",
             38        "__signature__": "426f662f9fe3b3533d9ce7f9dcf8af77",
             39        "__module__": "json_myobj", "__class__": "MyObj"}]
             40     ''',
             41
             42     # careful!
             43     '''
             44     [{"path": "/etc/passwd",
             45        "__signature__": "426f662f9fe3b3533d9ce7f9dcf8af77",
             46        "__module__": "os", "__class__": "unlink"}]
             47     ''',
             48     ]:




Sunday, March 20, 2011
json
             49          try:
             50              myobj_instance = json.loads(
             51                  encoded_object,
             52                  object_hook=dict_to_object,
             53                  )
             54              print myobj_instance
             55          except Exception as err:
             56              print 'ERROR:', err
             57          print




Sunday, March 20, 2011
json
       $ python hidden_stdlib/json_load_object_hook.py
       Loading "MyObj" from "json_myobj"
       Instantiating with {'s': u'instance value goes here'}
       [<MyObj(instance value goes here)>]


       ERROR: Invalid signature




Sunday, March 20, 2011
json
       $ python hidden_stdlib/json_load_object_hook.py
       Loading "MyObj" from "json_myobj"
       Instantiating with {'s': u'instance value goes here'}
       [<MyObj(instance value goes here)>]


       ERROR: Invalid signature




Sunday, March 20, 2011
Error Handling   sys.excepthook


Sunday, March 20, 2011
sys.excepthook
              6 def main():
              7     # do some work
              8     raise RuntimeError('Helpful error message')
              9
             10 if __name__ == '__main__':
             11     main()


       $ python hidden_stdlib/sys_excepthook_no_handler.py
       Traceback (most recent call last):
           File ".../hidden_stdlib/sys_excepthook_no_handler.py", line 11, in <module>
               main()
           File ".../hidden_stdlib/sys_excepthook_no_handler.py", line 8, in main
               raise RuntimeError('Helpful error message')
       RuntimeError: Helpful error message




Sunday, March 20, 2011
sys.excepthook
              6 import sys
              7
              8 def main():
              9     try:
             10          # do some work
             11          raise RuntimeError('Helpful error message')
             12     except Exception as err:
             13          sys.stderr.write('ERROR: %sn' % err)
             14
             15 if __name__ == '__main__':
             16     main()

       $ python hidden_stdlib/sys_excepthook_big_block.py
       ERROR: Helpful error message




Sunday, March 20, 2011
sys.excepthook
              6     import sys
              7
              8     def quiet_errors(exc_type, exc_value, traceback):
              9         sys.stderr.write('ERROR: %sn' % exc_value)
             10
             11     sys.excepthook = quiet_errors
             12
             13     def main():
             14         # do some work
             15         raise RuntimeError('Helpful error message')
             16
             17     if __name__ == '__main__':
             18         main()

       $ python hidden_stdlib/sys_excepthook.py
       ERROR: Helpful error message




Sunday, March 20, 2011
sys.excepthook
              6     import sys
              7
              8     def quiet_errors(exc_type, exc_value, traceback):
              9         sys.stderr.write('ERROR: %sn' % exc_value)
             10
             11     sys.excepthook = quiet_errors
             12
             13     def main():
             14         # do some work
             15         raise RuntimeError('Helpful error message')
             16
             17     if __name__ == '__main__':
             18         main()

       $ python hidden_stdlib/sys_excepthook.py
       ERROR: Helpful error message




Sunday, March 20, 2011
Communicating
                                         logging
                            with Users
Sunday, March 20, 2011
logging   loggers


Sunday, March 20, 2011
logging   loggers


Sunday, March 20, 2011
logging
             10     # Log verbosely
             11     root_logger = logging.getLogger('')
             12     root_logger.setLevel(logging.DEBUG)
             13
             14     # Set up console output to stderr
             15     console = logging.StreamHandler(sys.stderr)
             16     console_format = '%(message)s'
             17     console.setFormatter(logging.Formatter(console_format))
             18     console.setLevel(logging.INFO) # TODO: command line switch
             19     root_logger.addHandler(console)




Sunday, March 20, 2011
logging
             10     # Log verbosely
             11     root_logger = logging.getLogger('')
             12     root_logger.setLevel(logging.DEBUG)
             13
             14     # Set up console output to stderr
             15     console = logging.StreamHandler(sys.stderr)
             16     console_format = '%(message)s'
             17     console.setFormatter(logging.Formatter(console_format))
             18     console.setLevel(logging.INFO) # TODO: command line switch
             19     root_logger.addHandler(console)




Sunday, March 20, 2011
logging
             10     # Log verbosely
             11     root_logger = logging.getLogger('')
             12     root_logger.setLevel(logging.DEBUG)
             13
             14     # Set up console output to stderr
             15     console = logging.StreamHandler(sys.stderr)
             16     console_format = '%(message)s'
             17     console.setFormatter(logging.Formatter(console_format))
             18     console.setLevel(logging.INFO) # TODO: command line switch
             19     root_logger.addHandler(console)




Sunday, March 20, 2011
logging
             10     # Log verbosely
             11     root_logger = logging.getLogger('')
             12     root_logger.setLevel(logging.DEBUG)
             13
             14     # Set up console output to stderr
             15     console = logging.StreamHandler(sys.stderr)
             16     console_format = '%(message)s'
             17     console.setFormatter(logging.Formatter(console_format))
             18     console.setLevel(logging.INFO) # TODO: command line switch
             19     root_logger.addHandler(console)




Sunday, March 20, 2011
logging
             10     # Log verbosely
             11     root_logger = logging.getLogger('')
             12     root_logger.setLevel(logging.DEBUG)
             13
             14     # Set up console output to stderr
             15     console = logging.StreamHandler(sys.stderr)
             16     console_format = '%(message)s'
             17     console.setFormatter(logging.Formatter(console_format))
             18     console.setLevel(logging.INFO) # TODO: command line switch
             19     root_logger.addHandler(console)




Sunday, March 20, 2011
logging
         21 # Include debug messages when logging to a file
         22 file_handler = logging.handlers.RotatingFileHandler(
         23     'logging_example.log', # use a full path
         24     )
         25 file_format = '%(asctime)s %(levelname)6s %(name)s %
      (message)s'
         26 file_handler.setFormatter(logging.Formatter(file_format))
         27 file_handler.setLevel(logging.DEBUG)
         28 root_logger.addHandler(file_handler)




Sunday, March 20, 2011
logging
         21 # Include debug messages when logging to a file
         22 file_handler = logging.handlers.RotatingFileHandler(
         23     'logging_example.log', # use a full path
         24     )
         25 file_format = '%(asctime)s %(levelname)6s %(name)s %
      (message)s'
         26 file_handler.setFormatter(logging.Formatter(file_format))
         27 file_handler.setLevel(logging.DEBUG)
         28 root_logger.addHandler(file_handler)




Sunday, March 20, 2011
logging
         21 # Include debug messages when logging to a file
         22 file_handler = logging.handlers.RotatingFileHandler(
         23     'logging_example.log', # use a full path
         24     )
         25 file_format = '%(asctime)s %(levelname)6s %(name)s %
      (message)s'
         26 file_handler.setFormatter(logging.Formatter(file_format))
         27 file_handler.setLevel(logging.DEBUG)
         28 root_logger.addHandler(file_handler)




Sunday, March 20, 2011
logging
             30     # Log sample messages with different levels
             31     log = logging.getLogger(__name__)
             32     log.info('on the console and in the file')
             33     log.debug('only in the file')
             34     log.error('simple error message')
             35
             36     # Replace excepthook with logger
             37     def log_exception(exc_type, exc_value, traceback):
             38         logging.getLogger(__name__).error(exc_value)
             39     sys.excepthook = log_exception
             40
             41     # Send exceptions to the logger automatically
             42     raise RuntimeError('failure message')




Sunday, March 20, 2011
logging
             30     # Log sample messages with different levels
             31     log = logging.getLogger(__name__)
             32     log.info('on the console and in the file')
             33     log.debug('only in the file')
             34     log.error('simple error message')
             35
             36     # Replace excepthook with logger
             37     def log_exception(exc_type, exc_value, traceback):
             38         logging.getLogger(__name__).error(exc_value)
             39     sys.excepthook = log_exception
             40
             41     # Send exceptions to the logger automatically
             42     raise RuntimeError('failure message')




Sunday, March 20, 2011
logging
             30     # Log sample messages with different levels
             31     log = logging.getLogger(__name__)
             32     log.info('on the console and in the file')
             33     log.debug('only in the file')
             34     log.error('simple error message')
             35
             36     # Replace excepthook with logger
             37     def log_exception(exc_type, exc_value, traceback):
             38         logging.getLogger(__name__).error(exc_value)
             39     sys.excepthook = log_exception
             40
             41     # Send exceptions to the logger automatically
             42     raise RuntimeError('failure message')




Sunday, March 20, 2011
logging
       $ python hidden_stdlib/logging_example.py
       on the console and in the file
       simple error message
       failure message




Sunday, March 20, 2011
logging
       $ python hidden_stdlib/logging_example.py
       on the console and in the file
       simple error message
       failure message


       $ cat logging_example.log
       2011-02-13 11:28:30,036      INFO __main__ on the console and in
       the file
       2011-02-13 11:28:30,036     DEBUG __main__ only in the file
       2011-02-13 11:28:30,036     ERROR __main__ simple error message
       2011-02-13 11:28:30,036     ERROR __main__ failure message




Sunday, March 20, 2011
logging
       $ python hidden_stdlib/logging_example.py
       on the console and in the file
       simple error message
       failure message


       $ cat logging_example.log
       2011-02-13 11:28:30,036      INFO __main__ on the console and in
       the file
       2011-02-13 11:28:30,036     DEBUG __main__ only in the file
       2011-02-13 11:28:30,036     ERROR __main__ simple error message
       2011-02-13 11:28:30,036     ERROR __main__ failure message




Sunday, March 20, 2011
logging
       $ python hidden_stdlib/logging_example.py
       on the console and in the file
       simple error message
       failure message


       $ cat logging_example.log
       2011-02-13 11:28:30,036      INFO __main__ on the console and in
       the file
       2011-02-13 11:28:30,036     DEBUG __main__ only in the file
       2011-02-13 11:28:30,036     ERROR __main__ simple error message
       2011-02-13 11:28:30,036     ERROR __main__ failure message




Sunday, March 20, 2011
logging
       $ python hidden_stdlib/logging_example.py
       on the console and in the file
       simple error message
       failure message


       $ cat logging_example.log
       2011-02-13 11:28:30,036      INFO __main__ on the console and in
       the file
       2011-02-13 11:28:30,036     DEBUG __main__ only in the file
       2011-02-13 11:28:30,036     ERROR __main__ simple error message
       2011-02-13 11:28:30,036     ERROR __main__ failure message




Sunday, March 20, 2011
logging   Traceback Handler


Sunday, March 20, 2011
logging
             36     # Tracebacks should only go to the file
             37     traceback_log = logging.getLogger('traceback')
             38     traceback_log.propagate = False
             39     traceback_log.setLevel(logging.ERROR)
             40     traceback_log.addHandler(file_handler)
             41
             42     # Replace excepthook with logger
             43     def log_exception(exc_type, exc_value, traceback):
             44         logging.getLogger(__name__).error(exc_value)
             45         logging.getLogger('traceback').error(
             46             exc_value,
             47             exc_info=(exc_type, exc_value, traceback),
             48             )
             49     sys.excepthook = log_exception
             50
             51     # Send exceptions to the logger automatically
             52     raise RuntimeError('failure message')




Sunday, March 20, 2011
logging
             36     # Tracebacks should only go to the file
             37     traceback_log = logging.getLogger('traceback')
             38     traceback_log.propagate = False
             39     traceback_log.setLevel(logging.ERROR)
             40     traceback_log.addHandler(file_handler)
             41
             42     # Replace excepthook with logger
             43     def log_exception(exc_type, exc_value, traceback):
             44         logging.getLogger(__name__).error(exc_value)
             45         logging.getLogger('traceback').error(
             46             exc_value,
             47             exc_info=(exc_type, exc_value, traceback),
             48             )
             49     sys.excepthook = log_exception
             50
             51     # Send exceptions to the logger automatically
             52     raise RuntimeError('failure message')




Sunday, March 20, 2011
logging
       $ python hidden_stdlib/logging_example_tracebacks.py
       on the console and in the file
       simple error message
       failure message


       $ cat logging_example.log
       2011-02-13 11:33:22,592      INFO __main__ on the console and in the file
       2011-02-13 11:33:22,592     DEBUG __main__ only in the file
       2011-02-13 11:33:22,592     ERROR __main__ simple error message
       2011-02-13 11:33:22,592     ERROR __main__ failure message
       2011-02-13 11:33:22,593     ERROR traceback failure message
       Traceback (most recent call last):
           File "hidden_stdlib/logging_example_tracebacks.py", line 52, in <module>
               raise RuntimeError('failure message')
       RuntimeError: failure message




Sunday, March 20, 2011
More Information

      • http://docs.python.org/


      • http://www.doughellmann.com/PyMOTW/


      • https://bitbucket.org/dhellmann/hidden_stdlib/


      • The Python Standard Library By Example, Doug Hellmann


      • Python Essential Reference, David Beazley




Sunday, March 20, 2011
Image Credits

          2. http://www.uottawa.ca.libguides.com/content.php?pid=14825&sid=117658
          4. http://www.flickr.com/photos/seditiouscanary/1279041211/
          5. http://www.flickr.com/photos/elrentaplats/4902419885/in/photostream/
          6. http://www.flickr.com/photos/elrentaplats/4903005444/in/photostream/
          18. http://etc.usf.edu/clipart/58000/58063/58063_column_types.htm
          34. http://www.flickr.com/photos/akeg/3077866744/
          44. http://www.flickr.com/photos/yum9me/2807475045/
          64. http://commons.wikimedia.org/wiki/File:Zeichen_trissturis_error101.png
          72. http://www.flickr.com/photos/myklroventine/3545127104/in/
          faves-40068198@N04/




Sunday, March 20, 2011

Mais conteúdo relacionado

Mais procurados

The Ring programming language version 1.3 book - Part 19 of 88
The Ring programming language version 1.3 book - Part 19 of 88The Ring programming language version 1.3 book - Part 19 of 88
The Ring programming language version 1.3 book - Part 19 of 88Mahmoud Samir Fayed
 
Using Scala Slick at FortyTwo
Using Scala Slick at FortyTwoUsing Scala Slick at FortyTwo
Using Scala Slick at FortyTwoEishay Smith
 
Custom YUI Widgets
Custom YUI WidgetsCustom YUI Widgets
Custom YUI Widgetscyrildoussin
 
Jython: Python para la plataforma Java (EL2009)
Jython: Python para la plataforma Java (EL2009)Jython: Python para la plataforma Java (EL2009)
Jython: Python para la plataforma Java (EL2009)Leonardo Soto
 
Jython: Python para la plataforma Java (JRSL 09)
Jython: Python para la plataforma Java (JRSL 09)Jython: Python para la plataforma Java (JRSL 09)
Jython: Python para la plataforma Java (JRSL 09)Leonardo Soto
 
The Ring programming language version 1.10 book - Part 37 of 212
The Ring programming language version 1.10 book - Part 37 of 212The Ring programming language version 1.10 book - Part 37 of 212
The Ring programming language version 1.10 book - Part 37 of 212Mahmoud Samir Fayed
 
A evolução da persistência de dados (com sqlite) no android
A evolução da persistência de dados (com sqlite) no androidA evolução da persistência de dados (com sqlite) no android
A evolução da persistência de dados (com sqlite) no androidRodrigo de Souza Castro
 
Brief introduction of Slick
Brief introduction of SlickBrief introduction of Slick
Brief introduction of SlickKnoldus Inc.
 
BDD - Behavior Driven Development Webapps mit Groovy Spock und Geb
BDD - Behavior Driven Development Webapps mit Groovy Spock und GebBDD - Behavior Driven Development Webapps mit Groovy Spock und Geb
BDD - Behavior Driven Development Webapps mit Groovy Spock und GebChristian Baranowski
 
The uniform interface is 42
The uniform interface is 42The uniform interface is 42
The uniform interface is 42Yevhen Bobrov
 
The Ring programming language version 1.8 book - Part 33 of 202
The Ring programming language version 1.8 book - Part 33 of 202The Ring programming language version 1.8 book - Part 33 of 202
The Ring programming language version 1.8 book - Part 33 of 202Mahmoud Samir Fayed
 
The Ring programming language version 1.5.3 book - Part 28 of 184
The Ring programming language version 1.5.3 book - Part 28 of 184The Ring programming language version 1.5.3 book - Part 28 of 184
The Ring programming language version 1.5.3 book - Part 28 of 184Mahmoud Samir Fayed
 
The Ring programming language version 1.9 book - Part 35 of 210
The Ring programming language version 1.9 book - Part 35 of 210The Ring programming language version 1.9 book - Part 35 of 210
The Ring programming language version 1.9 book - Part 35 of 210Mahmoud Samir Fayed
 

Mais procurados (20)

Web2py
Web2pyWeb2py
Web2py
 
The Ring programming language version 1.3 book - Part 19 of 88
The Ring programming language version 1.3 book - Part 19 of 88The Ring programming language version 1.3 book - Part 19 of 88
The Ring programming language version 1.3 book - Part 19 of 88
 
My sql tutorial-oscon-2012
My sql tutorial-oscon-2012My sql tutorial-oscon-2012
My sql tutorial-oscon-2012
 
Using Scala Slick at FortyTwo
Using Scala Slick at FortyTwoUsing Scala Slick at FortyTwo
Using Scala Slick at FortyTwo
 
Custom YUI Widgets
Custom YUI WidgetsCustom YUI Widgets
Custom YUI Widgets
 
JDBC
JDBCJDBC
JDBC
 
Jython: Python para la plataforma Java (EL2009)
Jython: Python para la plataforma Java (EL2009)Jython: Python para la plataforma Java (EL2009)
Jython: Python para la plataforma Java (EL2009)
 
H base programming
H base programmingH base programming
H base programming
 
Spock and Geb in Action
Spock and Geb in ActionSpock and Geb in Action
Spock and Geb in Action
 
Jython: Python para la plataforma Java (JRSL 09)
Jython: Python para la plataforma Java (JRSL 09)Jython: Python para la plataforma Java (JRSL 09)
Jython: Python para la plataforma Java (JRSL 09)
 
The Ring programming language version 1.10 book - Part 37 of 212
The Ring programming language version 1.10 book - Part 37 of 212The Ring programming language version 1.10 book - Part 37 of 212
The Ring programming language version 1.10 book - Part 37 of 212
 
A evolução da persistência de dados (com sqlite) no android
A evolução da persistência de dados (com sqlite) no androidA evolução da persistência de dados (com sqlite) no android
A evolução da persistência de dados (com sqlite) no android
 
Brief introduction of Slick
Brief introduction of SlickBrief introduction of Slick
Brief introduction of Slick
 
BDD - Behavior Driven Development Webapps mit Groovy Spock und Geb
BDD - Behavior Driven Development Webapps mit Groovy Spock und GebBDD - Behavior Driven Development Webapps mit Groovy Spock und Geb
BDD - Behavior Driven Development Webapps mit Groovy Spock und Geb
 
How te bring common UI patterns to ADF
How te bring common UI patterns to ADFHow te bring common UI patterns to ADF
How te bring common UI patterns to ADF
 
The uniform interface is 42
The uniform interface is 42The uniform interface is 42
The uniform interface is 42
 
Spock and Geb
Spock and GebSpock and Geb
Spock and Geb
 
The Ring programming language version 1.8 book - Part 33 of 202
The Ring programming language version 1.8 book - Part 33 of 202The Ring programming language version 1.8 book - Part 33 of 202
The Ring programming language version 1.8 book - Part 33 of 202
 
The Ring programming language version 1.5.3 book - Part 28 of 184
The Ring programming language version 1.5.3 book - Part 28 of 184The Ring programming language version 1.5.3 book - Part 28 of 184
The Ring programming language version 1.5.3 book - Part 28 of 184
 
The Ring programming language version 1.9 book - Part 35 of 210
The Ring programming language version 1.9 book - Part 35 of 210The Ring programming language version 1.9 book - Part 35 of 210
The Ring programming language version 1.9 book - Part 35 of 210
 

Semelhante a Hidden Treasures of the Python Standard Library

Building Apps with MongoDB
Building Apps with MongoDBBuilding Apps with MongoDB
Building Apps with MongoDBNate Abele
 
Spock: Test Well and Prosper
Spock: Test Well and ProsperSpock: Test Well and Prosper
Spock: Test Well and ProsperKen Kousen
 
Declarative Data Modeling in Python
Declarative Data Modeling in PythonDeclarative Data Modeling in Python
Declarative Data Modeling in PythonJoshua Forman
 
2012-08-29 - NoSQL Bootcamp (Redis, RavenDB & MongoDB für .NET Entwickler)
2012-08-29 - NoSQL Bootcamp (Redis, RavenDB & MongoDB für .NET Entwickler)2012-08-29 - NoSQL Bootcamp (Redis, RavenDB & MongoDB für .NET Entwickler)
2012-08-29 - NoSQL Bootcamp (Redis, RavenDB & MongoDB für .NET Entwickler)Johannes Hoppe
 
M.TECH 1ST SEM COMPUTER SCIENCE ADBMS LAB PROGRAMS
M.TECH 1ST SEM COMPUTER SCIENCE ADBMS LAB PROGRAMSM.TECH 1ST SEM COMPUTER SCIENCE ADBMS LAB PROGRAMS
M.TECH 1ST SEM COMPUTER SCIENCE ADBMS LAB PROGRAMSSupriya Radhakrishna
 
Indexing & query optimization
Indexing & query optimizationIndexing & query optimization
Indexing & query optimizationJared Rosoff
 
Introduction to Active Record - Silicon Valley Ruby Conference 2007
Introduction to Active Record - Silicon Valley Ruby Conference 2007Introduction to Active Record - Silicon Valley Ruby Conference 2007
Introduction to Active Record - Silicon Valley Ruby Conference 2007Rabble .
 
Rapid and Scalable Development with MongoDB, PyMongo, and Ming
Rapid and Scalable Development with MongoDB, PyMongo, and MingRapid and Scalable Development with MongoDB, PyMongo, and Ming
Rapid and Scalable Development with MongoDB, PyMongo, and MingRick Copeland
 
MongoDB-MFF.pptx
MongoDB-MFF.pptxMongoDB-MFF.pptx
MongoDB-MFF.pptxYellowGem
 
Nosql hands on handout 04
Nosql hands on handout 04Nosql hands on handout 04
Nosql hands on handout 04Krishna Sankar
 
Enable Database Service over HTTP or IBM WebSphere MQ in 15_minutes with IAS
Enable Database Service over HTTP or IBM WebSphere MQ in 15_minutes with IASEnable Database Service over HTTP or IBM WebSphere MQ in 15_minutes with IAS
Enable Database Service over HTTP or IBM WebSphere MQ in 15_minutes with IASInvenire Aude
 
Slides python elixir
Slides python elixirSlides python elixir
Slides python elixirAdel Totott
 
MySQL flexible schema and JSON for Internet of Things
MySQL flexible schema and JSON for Internet of ThingsMySQL flexible schema and JSON for Internet of Things
MySQL flexible schema and JSON for Internet of ThingsAlexander Rubin
 
Introduction to Active Record at MySQL Conference 2007
Introduction to Active Record at MySQL Conference 2007Introduction to Active Record at MySQL Conference 2007
Introduction to Active Record at MySQL Conference 2007Rabble .
 
Getting started with Clojure
Getting started with ClojureGetting started with Clojure
Getting started with ClojureJohn Stevenson
 
San Francisco Java User Group
San Francisco Java User GroupSan Francisco Java User Group
San Francisco Java User Groupkchodorow
 
PostgreSQL Open SV 2018
PostgreSQL Open SV 2018PostgreSQL Open SV 2018
PostgreSQL Open SV 2018artgillespie
 

Semelhante a Hidden Treasures of the Python Standard Library (20)

Latinoware
LatinowareLatinoware
Latinoware
 
Building Apps with MongoDB
Building Apps with MongoDBBuilding Apps with MongoDB
Building Apps with MongoDB
 
Spl Not A Bridge Too Far phpNW09
Spl Not A Bridge Too Far phpNW09Spl Not A Bridge Too Far phpNW09
Spl Not A Bridge Too Far phpNW09
 
Spock: Test Well and Prosper
Spock: Test Well and ProsperSpock: Test Well and Prosper
Spock: Test Well and Prosper
 
Declarative Data Modeling in Python
Declarative Data Modeling in PythonDeclarative Data Modeling in Python
Declarative Data Modeling in Python
 
2012-08-29 - NoSQL Bootcamp (Redis, RavenDB & MongoDB für .NET Entwickler)
2012-08-29 - NoSQL Bootcamp (Redis, RavenDB & MongoDB für .NET Entwickler)2012-08-29 - NoSQL Bootcamp (Redis, RavenDB & MongoDB für .NET Entwickler)
2012-08-29 - NoSQL Bootcamp (Redis, RavenDB & MongoDB für .NET Entwickler)
 
M.TECH 1ST SEM COMPUTER SCIENCE ADBMS LAB PROGRAMS
M.TECH 1ST SEM COMPUTER SCIENCE ADBMS LAB PROGRAMSM.TECH 1ST SEM COMPUTER SCIENCE ADBMS LAB PROGRAMS
M.TECH 1ST SEM COMPUTER SCIENCE ADBMS LAB PROGRAMS
 
Indexing & query optimization
Indexing & query optimizationIndexing & query optimization
Indexing & query optimization
 
Introduction to Active Record - Silicon Valley Ruby Conference 2007
Introduction to Active Record - Silicon Valley Ruby Conference 2007Introduction to Active Record - Silicon Valley Ruby Conference 2007
Introduction to Active Record - Silicon Valley Ruby Conference 2007
 
Rapid and Scalable Development with MongoDB, PyMongo, and Ming
Rapid and Scalable Development with MongoDB, PyMongo, and MingRapid and Scalable Development with MongoDB, PyMongo, and Ming
Rapid and Scalable Development with MongoDB, PyMongo, and Ming
 
Advanced Django
Advanced DjangoAdvanced Django
Advanced Django
 
MongoDB-MFF.pptx
MongoDB-MFF.pptxMongoDB-MFF.pptx
MongoDB-MFF.pptx
 
Nosql hands on handout 04
Nosql hands on handout 04Nosql hands on handout 04
Nosql hands on handout 04
 
Enable Database Service over HTTP or IBM WebSphere MQ in 15_minutes with IAS
Enable Database Service over HTTP or IBM WebSphere MQ in 15_minutes with IASEnable Database Service over HTTP or IBM WebSphere MQ in 15_minutes with IAS
Enable Database Service over HTTP or IBM WebSphere MQ in 15_minutes with IAS
 
Slides python elixir
Slides python elixirSlides python elixir
Slides python elixir
 
MySQL flexible schema and JSON for Internet of Things
MySQL flexible schema and JSON for Internet of ThingsMySQL flexible schema and JSON for Internet of Things
MySQL flexible schema and JSON for Internet of Things
 
Introduction to Active Record at MySQL Conference 2007
Introduction to Active Record at MySQL Conference 2007Introduction to Active Record at MySQL Conference 2007
Introduction to Active Record at MySQL Conference 2007
 
Getting started with Clojure
Getting started with ClojureGetting started with Clojure
Getting started with Clojure
 
San Francisco Java User Group
San Francisco Java User GroupSan Francisco Java User Group
San Francisco Java User Group
 
PostgreSQL Open SV 2018
PostgreSQL Open SV 2018PostgreSQL Open SV 2018
PostgreSQL Open SV 2018
 

Mais de doughellmann

Reno: A new way to manage release notes
Reno: A new way to manage release notesReno: A new way to manage release notes
Reno: A new way to manage release notesdoughellmann
 
Reno A New Way to Manage Release Notes
Reno   A New Way to Manage Release NotesReno   A New Way to Manage Release Notes
Reno A New Way to Manage Release Notesdoughellmann
 
How OpenStack Makes Python Better (and vice-versa)
How OpenStack Makes Python Better (and vice-versa)How OpenStack Makes Python Better (and vice-versa)
How OpenStack Makes Python Better (and vice-versa)doughellmann
 
How I Built a Power Debugger Out of the Standard Library and Things I Found o...
How I Built a Power Debugger Out of the Standard Library and Things I Found o...How I Built a Power Debugger Out of the Standard Library and Things I Found o...
How I Built a Power Debugger Out of the Standard Library and Things I Found o...doughellmann
 
Rolling with the Times: Using wheels, pbr, and Twine for Distributing and Ins...
Rolling with the Times: Using wheels, pbr, and Twine for Distributing and Ins...Rolling with the Times: Using wheels, pbr, and Twine for Distributing and Ins...
Rolling with the Times: Using wheels, pbr, and Twine for Distributing and Ins...doughellmann
 
Herding cats into boxes
Herding cats into boxesHerding cats into boxes
Herding cats into boxesdoughellmann
 
How I Built a Power Debugger Out of the Standard Library and Things I Found o...
How I Built a Power Debugger Out of the Standard Library and Things I Found o...How I Built a Power Debugger Out of the Standard Library and Things I Found o...
How I Built a Power Debugger Out of the Standard Library and Things I Found o...doughellmann
 
OpenStack 5th Birthday
OpenStack 5th BirthdayOpenStack 5th Birthday
OpenStack 5th Birthdaydoughellmann
 
Regexes and-performance-testing
Regexes and-performance-testingRegexes and-performance-testing
Regexes and-performance-testingdoughellmann
 
OpenStack Atlanta-2014-12-18
OpenStack Atlanta-2014-12-18OpenStack Atlanta-2014-12-18
OpenStack Atlanta-2014-12-18doughellmann
 
Taking the Long View: How the Oslo Program Reduces Technical Debt
Taking the Long View: How the Oslo Program Reduces Technical DebtTaking the Long View: How the Oslo Program Reduces Technical Debt
Taking the Long View: How the Oslo Program Reduces Technical Debtdoughellmann
 
Oslo Program Overview, OpenStack Atlanta
Oslo Program Overview, OpenStack AtlantaOslo Program Overview, OpenStack Atlanta
Oslo Program Overview, OpenStack Atlantadoughellmann
 
Dynamic Code Patterns: Extending Your Applications with Plugins
Dynamic Code Patterns: Extending Your Applications with PluginsDynamic Code Patterns: Extending Your Applications with Plugins
Dynamic Code Patterns: Extending Your Applications with Pluginsdoughellmann
 
Better Documentation Through Automation: Creating docutils & Sphinx Extensions
Better Documentation Through Automation: Creating docutils & Sphinx ExtensionsBetter Documentation Through Automation: Creating docutils & Sphinx Extensions
Better Documentation Through Automation: Creating docutils & Sphinx Extensionsdoughellmann
 
An Introduction to the Zen of Python
An Introduction to the Zen of PythonAn Introduction to the Zen of Python
An Introduction to the Zen of Pythondoughellmann
 

Mais de doughellmann (15)

Reno: A new way to manage release notes
Reno: A new way to manage release notesReno: A new way to manage release notes
Reno: A new way to manage release notes
 
Reno A New Way to Manage Release Notes
Reno   A New Way to Manage Release NotesReno   A New Way to Manage Release Notes
Reno A New Way to Manage Release Notes
 
How OpenStack Makes Python Better (and vice-versa)
How OpenStack Makes Python Better (and vice-versa)How OpenStack Makes Python Better (and vice-versa)
How OpenStack Makes Python Better (and vice-versa)
 
How I Built a Power Debugger Out of the Standard Library and Things I Found o...
How I Built a Power Debugger Out of the Standard Library and Things I Found o...How I Built a Power Debugger Out of the Standard Library and Things I Found o...
How I Built a Power Debugger Out of the Standard Library and Things I Found o...
 
Rolling with the Times: Using wheels, pbr, and Twine for Distributing and Ins...
Rolling with the Times: Using wheels, pbr, and Twine for Distributing and Ins...Rolling with the Times: Using wheels, pbr, and Twine for Distributing and Ins...
Rolling with the Times: Using wheels, pbr, and Twine for Distributing and Ins...
 
Herding cats into boxes
Herding cats into boxesHerding cats into boxes
Herding cats into boxes
 
How I Built a Power Debugger Out of the Standard Library and Things I Found o...
How I Built a Power Debugger Out of the Standard Library and Things I Found o...How I Built a Power Debugger Out of the Standard Library and Things I Found o...
How I Built a Power Debugger Out of the Standard Library and Things I Found o...
 
OpenStack 5th Birthday
OpenStack 5th BirthdayOpenStack 5th Birthday
OpenStack 5th Birthday
 
Regexes and-performance-testing
Regexes and-performance-testingRegexes and-performance-testing
Regexes and-performance-testing
 
OpenStack Atlanta-2014-12-18
OpenStack Atlanta-2014-12-18OpenStack Atlanta-2014-12-18
OpenStack Atlanta-2014-12-18
 
Taking the Long View: How the Oslo Program Reduces Technical Debt
Taking the Long View: How the Oslo Program Reduces Technical DebtTaking the Long View: How the Oslo Program Reduces Technical Debt
Taking the Long View: How the Oslo Program Reduces Technical Debt
 
Oslo Program Overview, OpenStack Atlanta
Oslo Program Overview, OpenStack AtlantaOslo Program Overview, OpenStack Atlanta
Oslo Program Overview, OpenStack Atlanta
 
Dynamic Code Patterns: Extending Your Applications with Plugins
Dynamic Code Patterns: Extending Your Applications with PluginsDynamic Code Patterns: Extending Your Applications with Plugins
Dynamic Code Patterns: Extending Your Applications with Plugins
 
Better Documentation Through Automation: Creating docutils & Sphinx Extensions
Better Documentation Through Automation: Creating docutils & Sphinx ExtensionsBetter Documentation Through Automation: Creating docutils & Sphinx Extensions
Better Documentation Through Automation: Creating docutils & Sphinx Extensions
 
An Introduction to the Zen of Python
An Introduction to the Zen of PythonAn Introduction to the Zen of Python
An Introduction to the Zen of Python
 

Último

Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1DianaGray10
 
The Kubernetes Gateway API and its role in Cloud Native API Management
The Kubernetes Gateway API and its role in Cloud Native API ManagementThe Kubernetes Gateway API and its role in Cloud Native API Management
The Kubernetes Gateway API and its role in Cloud Native API ManagementNuwan Dias
 
Computer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsComputer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsSeth Reyes
 
UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8DianaGray10
 
100+ ChatGPT Prompts for SEO Optimization
100+ ChatGPT Prompts for SEO Optimization100+ ChatGPT Prompts for SEO Optimization
100+ ChatGPT Prompts for SEO Optimizationarrow10202532yuvraj
 
VoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBXVoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBXTarek Kalaji
 
Nanopower In Semiconductor Industry.pdf
Nanopower  In Semiconductor Industry.pdfNanopower  In Semiconductor Industry.pdf
Nanopower In Semiconductor Industry.pdfPedro Manuel
 
UiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPathCommunity
 
20230202 - Introduction to tis-py
20230202 - Introduction to tis-py20230202 - Introduction to tis-py
20230202 - Introduction to tis-pyJamie (Taka) Wang
 
99.99% of Your Traces Are (Probably) Trash (SRECon NA 2024).pdf
99.99% of Your Traces  Are (Probably) Trash (SRECon NA 2024).pdf99.99% of Your Traces  Are (Probably) Trash (SRECon NA 2024).pdf
99.99% of Your Traces Are (Probably) Trash (SRECon NA 2024).pdfPaige Cruz
 
Governance in SharePoint Premium:What's in the box?
Governance in SharePoint Premium:What's in the box?Governance in SharePoint Premium:What's in the box?
Governance in SharePoint Premium:What's in the box?Juan Carlos Gonzalez
 
UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6DianaGray10
 
Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024D Cloud Solutions
 
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDEADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDELiveplex
 
Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )Brian Pichman
 
Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Adtran
 
Comparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioComparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioChristian Posta
 
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfUiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfDianaGray10
 
OpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureOpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureEric D. Schabell
 

Último (20)

Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1
 
The Kubernetes Gateway API and its role in Cloud Native API Management
The Kubernetes Gateway API and its role in Cloud Native API ManagementThe Kubernetes Gateway API and its role in Cloud Native API Management
The Kubernetes Gateway API and its role in Cloud Native API Management
 
Computer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsComputer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and Hazards
 
UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8
 
100+ ChatGPT Prompts for SEO Optimization
100+ ChatGPT Prompts for SEO Optimization100+ ChatGPT Prompts for SEO Optimization
100+ ChatGPT Prompts for SEO Optimization
 
VoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBXVoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBX
 
Nanopower In Semiconductor Industry.pdf
Nanopower  In Semiconductor Industry.pdfNanopower  In Semiconductor Industry.pdf
Nanopower In Semiconductor Industry.pdf
 
UiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation Developers
 
20230202 - Introduction to tis-py
20230202 - Introduction to tis-py20230202 - Introduction to tis-py
20230202 - Introduction to tis-py
 
99.99% of Your Traces Are (Probably) Trash (SRECon NA 2024).pdf
99.99% of Your Traces  Are (Probably) Trash (SRECon NA 2024).pdf99.99% of Your Traces  Are (Probably) Trash (SRECon NA 2024).pdf
99.99% of Your Traces Are (Probably) Trash (SRECon NA 2024).pdf
 
Governance in SharePoint Premium:What's in the box?
Governance in SharePoint Premium:What's in the box?Governance in SharePoint Premium:What's in the box?
Governance in SharePoint Premium:What's in the box?
 
UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6
 
20230104 - machine vision
20230104 - machine vision20230104 - machine vision
20230104 - machine vision
 
Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024
 
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDEADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
 
Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )
 
Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™
 
Comparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioComparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and Istio
 
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfUiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
 
OpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureOpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability Adventure
 

Hidden Treasures of the Python Standard Library

  • 1. Hidden Treasures of the Standard Library Doug Hellmann PyCon February, 2011 Sunday, March 20, 2011
  • 2. Python Module of the Week: “I read the docs, so you don’t have to.” Sunday, March 20, 2011
  • 3. Read This! Pre-order now Sunday, March 20, 2011
  • 4. Goals Usable Tips Sunday, March 20, 2011
  • 5. Look Around Change Focus Sunday, March 20, 2011
  • 6. Look Around 100+ Modules Sunday, March 20, 2011
  • 7. Full Name User Name Email Doug Hellmann dhellmann doug.hellmann@gmail.com Guest account, guest guest@example.com no login Parsing Structured Data csv dialects Sunday, March 20, 2011
  • 8. Test Database 1 #!/bin/sh 2 3 rm -f test.db 4 5 sqlite3 test.db <<EOF 6 7 CREATE TABLE users ( 8 fullname text, 9 username text, 10 email text 11 ); 12 13 INSERT INTO users (fullname, username, email) 14 VALUES ('Doug Hellmann', 'dhellmann', 'doug.hellmann@gmail.com'); 15 16 INSERT INTO users (fullname, username, email) 17 VALUES ('Guest account, no login', 'guest', 'guest@example.com'); 18 19 EOF 20 Sunday, March 20, 2011
  • 9. Test Database 1 #!/bin/sh 2 3 rm -f test.db 4 5 sqlite3 test.db <<EOF 6 7 CREATE TABLE users ( 8 fullname text, 9 username text, 10 email text 11 ); 12 13 INSERT INTO users (fullname, username, email) 14 VALUES ('Doug Hellmann', 'dhellmann', 'doug.hellmann@gmail.com'); 15 16 INSERT INTO users (fullname, username, email) 17 VALUES ('Guest account, no login', 'guest', 'guest@example.com'); 18 19 EOF 20 Sunday, March 20, 2011
  • 10. Database Contents $ ./hidden_stdlib/csv_mkdb.sh $ sqlite3 -noheader test.db 'select * from users' Doug Hellmann|dhellmann|doug.hellmann@gmail.com Guest account, no login|guest|guest@example.com Sunday, March 20, 2011
  • 11. Database Contents $ ./hidden_stdlib/csv_mkdb.sh $ sqlite3 -noheader test.db 'select * from users' Doug Hellmann|dhellmann|doug.hellmann@gmail.com Guest account, no login|guest|guest@example.com Sunday, March 20, 2011
  • 12. csv Dialect 9 csv.register_dialect('pipes', delimiter='|') 10 11 reader = csv.reader(sys.stdin, dialect='pipes') 12 for row in reader: 13 print row Sunday, March 20, 2011
  • 13. csv Dialect 9 csv.register_dialect('pipes', delimiter='|') 10 11 reader = csv.reader(sys.stdin, dialect='pipes') 12 for row in reader: 13 print row Sunday, March 20, 2011
  • 14. csv Dialect 9 csv.register_dialect('pipes', delimiter='|') 10 11 reader = csv.reader(sys.stdin, dialect='pipes') 12 for row in reader: 13 print row Sunday, March 20, 2011
  • 15. Database Contents $ ./hidden_stdlib/csv_mkdb.sh $ sqlite3 -noheader test.db 'select * from users' Doug Hellmann|dhellmann|doug.hellmann@gmail.com Guest account, no login|guest|guest@example.com $ sqlite3 -noheader test.db 'select * from users' | python ./hidden_stdlib/csv_pipes.py ['Doug Hellmann', 'dhellmann', 'doug.hellmann@gmail.com'] ['Guest account, no login', 'guest', 'guest@example.com'] Sunday, March 20, 2011
  • 16. Database Contents $ ./hidden_stdlib/csv_mkdb.sh $ sqlite3 -noheader test.db 'select * from users' Doug Hellmann|dhellmann|doug.hellmann@gmail.com Guest account, no login|guest|guest@example.com $ sqlite3 -noheader test.db 'select * from users' | python ./hidden_stdlib/csv_pipes.py ['Doug Hellmann', 'dhellmann', 'doug.hellmann@gmail.com'] ['Guest account, no login', 'guest', 'guest@example.com'] Sunday, March 20, 2011
  • 17. Custom Column Types sqlite3 Sunday, March 20, 2011
  • 18. Custom SQLite Column Types 42 # Insert the objects into the database 43 print 'Inserting:' 44 to_save = [ (MyObj('this is a value to save'),), 45 (MyObj(42),), 46 ] 47 cursor.executemany( 48 "insert into obj (data) values (?)", 49 to_save) 50 51 # Query the database for the objects just saved 52 print 'nQuerying:' 53 cursor.execute("select id, data from obj") 54 for obj_id, obj in cursor.fetchall(): 55 print 'Retrieved', obj_id, type(obj), obj Sunday, March 20, 2011
  • 19. Custom SQLite Column Types 42 # Insert the objects into the database 43 print 'Inserting:' 44 to_save = [ (MyObj('this is a value to save'),), 45 (MyObj(42),), 46 ] 47 cursor.executemany( 48 "insert into obj (data) values (?)", 49 to_save) 50 51 # Query the database for the objects just saved 52 print 'nQuerying:' 53 cursor.execute("select id, data from obj") 54 for obj_id, obj in cursor.fetchall(): 55 print 'Retrieved', obj_id, type(obj), obj Sunday, March 20, 2011
  • 20. Custom SQLite Column Types 42 # Insert the objects into the database 43 print 'Inserting:' 44 to_save = [ (MyObj('this is a value to save'),), 45 (MyObj(42),), 46 ] 47 cursor.executemany( 48 "insert into obj (data) values (?)", 49 to_save) 50 51 # Query the database for the objects just saved 52 print 'nQuerying:' 53 cursor.execute("select id, data from obj") 54 for obj_id, obj in cursor.fetchall(): 55 print 'Retrieved', obj_id, type(obj), obj Sunday, March 20, 2011
  • 21. Custom SQLite Column Types 9 class MyObj(object): 10 def __init__(self, arg): 11 self.arg = arg 12 def __str__(self): 13 return 'MyObj(%r)' % self.arg Sunday, March 20, 2011
  • 22. Custom SQLite Column Types 17 def adapter_func(obj): 18 """memory -> storage""" 19 print 'Saving:', obj 20 return pickle.dumps(obj) 21 22 sqlite3.register_adapter(MyObj, adapter_func) 23 24 def converter_func(data): 25 """storage -> memory""" 26 return pickle.loads(data) 27 28 sqlite3.register_converter("MyObj", converter_func) Sunday, March 20, 2011
  • 23. Custom SQLite Column Types 17 def adapter_func(obj): 18 """memory -> storage""" 19 print 'Saving:', obj 20 return pickle.dumps(obj) 21 22 sqlite3.register_adapter(MyObj, adapter_func) 23 24 def converter_func(data): 25 """storage -> memory""" 26 return pickle.loads(data) 27 28 sqlite3.register_converter("MyObj", converter_func) Sunday, March 20, 2011
  • 24. Custom SQLite Column Types 17 def adapter_func(obj): 18 """memory -> storage""" 19 print 'Saving:', obj 20 return pickle.dumps(obj) 21 22 sqlite3.register_adapter(MyObj, adapter_func) 23 24 def converter_func(data): 25 """storage -> memory""" 26 return pickle.loads(data) 27 28 sqlite3.register_converter("MyObj", converter_func) Sunday, March 20, 2011
  • 25. Custom SQLite Column Types 17 def adapter_func(obj): 18 """memory -> storage""" 19 print 'Saving:', obj 20 return pickle.dumps(obj) 21 22 sqlite3.register_adapter(MyObj, adapter_func) 23 24 def converter_func(data): 25 """storage -> memory""" 26 return pickle.loads(data) 27 28 sqlite3.register_converter("MyObj", converter_func) Sunday, March 20, 2011
  • 26. Custom SQLite Column Types 30 with sqlite3.connect( 31 'type_demo.db', 32 detect_types=sqlite3.PARSE_DECLTYPES) as conn: 33 # Create a table with column of type "MyObj" 34 conn.execute(""" 35 create table if not exists obj ( 36 id integer primary key autoincrement not null, 37 data MyObj 38 ) 39 """) Sunday, March 20, 2011
  • 27. Custom SQLite Column Types 30 with sqlite3.connect( 31 'type_demo.db', 32 detect_types=sqlite3.PARSE_DECLTYPES) as conn: 33 # Create a table with column of type "MyObj" 34 conn.execute(""" 35 create table if not exists obj ( 36 id integer primary key autoincrement not null, 37 data MyObj 38 ) 39 """) Sunday, March 20, 2011
  • 28. Custom SQLite Column Types $ python hidden_stdlib/sqlite3_custom_type.py Inserting: Saving: MyObj('this is a value to save') Saving: MyObj(42) Querying: Retrieved 1 <class '__main__.MyObj'> MyObj('this is a value to save') Retrieved 2 <class '__main__.MyObj'> MyObj(42) Sunday, March 20, 2011
  • 29. Custom SQLite Column Types $ python hidden_stdlib/sqlite3_custom_type.py Inserting: Saving: MyObj('this is a value to save') Saving: MyObj(42) Querying: Retrieved 1 <class '__main__.MyObj'> MyObj('this is a value to save') Retrieved 2 <class '__main__.MyObj'> MyObj(42) Sunday, March 20, 2011
  • 30. Custom SQLite Column Types $ python hidden_stdlib/sqlite3_custom_type.py Inserting: Saving: MyObj('this is a value to save') Saving: MyObj(42) Querying: Retrieved 1 <class '__main__.MyObj'> MyObj('this is a value to save') Retrieved 2 <class '__main__.MyObj'> MyObj(42) Sunday, March 20, 2011
  • 31. Signed Serialized Data hmac Sunday, March 20, 2011
  • 32. hmac 11 message = 'The message' 12 encoded_message = pickle.dumps(message) 13 14 digest_maker = hmac.new('shared-secret-value') 15 digest_maker.update(encoded_message) 16 signature = digest_maker.hexdigest() 17 18 print 'Outgoing signature:', signature 19 20 # Simulate sending the message 21 buffer = StringIO('%sn%dn%s' % (signature, 22 len(encoded_message), 23 encoded_message 24 )) Sunday, March 20, 2011
  • 33. hmac 11 message = 'The message' 12 encoded_message = pickle.dumps(message) 13 14 digest_maker = hmac.new('shared-secret-value') 15 digest_maker.update(encoded_message) 16 signature = digest_maker.hexdigest() 17 18 print 'Outgoing signature:', signature 19 20 # Simulate sending the message 21 buffer = StringIO('%sn%dn%s' % (signature, 22 len(encoded_message), 23 encoded_message 24 )) Sunday, March 20, 2011
  • 34. hmac 11 message = 'The message' 12 encoded_message = pickle.dumps(message) 13 14 digest_maker = hmac.new('shared-secret-value') 15 digest_maker.update(encoded_message) 16 signature = digest_maker.hexdigest() 17 18 print 'Outgoing signature:', signature 19 20 # Simulate sending the message 21 buffer = StringIO('%sn%dn%s' % (signature, 22 len(encoded_message), 23 encoded_message 24 )) Sunday, March 20, 2011
  • 35. hmac 26 # "Receive" the message 27 read_signature = buffer.readline().rstrip() 28 message_len = int(buffer.readline()) 29 read_message = buffer.read(message_len) 30 31 # Check the signature of the incoming data 32 digest_maker = hmac.new('shared-secret-value', 33 read_message) 34 computed_signature = digest_maker.hexdigest() 35 print 'Computed signature:', signature Sunday, March 20, 2011
  • 36. hmac 26 # "Receive" the message 27 read_signature = buffer.readline().rstrip() 28 message_len = int(buffer.readline()) 29 read_message = buffer.read(message_len) 30 31 # Check the signature of the incoming data 32 digest_maker = hmac.new('shared-secret-value', 33 read_message) 34 computed_signature = digest_maker.hexdigest() 35 print 'Computed signature:', signature Sunday, March 20, 2011
  • 37. hmac 37 if computed_signature == read_signature: 38 print 'nValid message, processed' 39 safe_message = pickle.loads(read_message) 40 print 'Message:', safe_message 41 else: 42 raise ValueError('Invalid message, discarded') Sunday, March 20, 2011
  • 38. hmac 37 if computed_signature == read_signature: 38 print 'nValid message, processed' 39 safe_message = pickle.loads(read_message) 40 print 'Message:', safe_message 41 else: 42 raise ValueError('Invalid message, discarded') Sunday, March 20, 2011
  • 39. hmac $ python hidden_stdlib/hmac_message_signing.py Outgoing signature: 3498b6fa42a75dc4603da6f4c67505a2 Computed signature: 3498b6fa42a75dc4603da6f4c67505a2 Valid message, processed Message: The message Sunday, March 20, 2011
  • 40. Serializing Objects json Sunday, March 20, 2011
  • 41. json 10 def convert_to_builtin_type(obj): 11 """object->dictionary""" 12 print 'convert_to_builtin_type(%r)' % obj 13 class_name = obj.__class__.__name__ 14 module_name = obj.__module__ 15 digest_maker = hmac.new('PyCon2011', 16 module_name + class_name) 17 signature = digest_maker.hexdigest() 18 d = { '__class__':class_name, 19 '__module__':module_name, 20 '__signature__':signature, 21 } 22 d.update(obj.__dict__) 23 return d 24 25 obj = json_myobj.MyObj('instance value goes here') 26 print json.dumps(obj, default=convert_to_builtin_type) Sunday, March 20, 2011
  • 42. json 10 def convert_to_builtin_type(obj): 11 """object->dictionary""" 12 print 'convert_to_builtin_type(%r)' % obj 13 class_name = obj.__class__.__name__ 14 module_name = obj.__module__ 15 digest_maker = hmac.new('PyCon2011', 16 module_name + class_name) 17 signature = digest_maker.hexdigest() 18 d = { '__class__':class_name, 19 '__module__':module_name, 20 '__signature__':signature, 21 } 22 d.update(obj.__dict__) 23 return d 24 25 obj = json_myobj.MyObj('instance value goes here') 26 print json.dumps(obj, default=convert_to_builtin_type) Sunday, March 20, 2011
  • 43. json 10 def convert_to_builtin_type(obj): 11 """object->dictionary""" 12 print 'convert_to_builtin_type(%r)' % obj 13 class_name = obj.__class__.__name__ 14 module_name = obj.__module__ 15 digest_maker = hmac.new('PyCon2011', 16 module_name + class_name) 17 signature = digest_maker.hexdigest() 18 d = { '__class__':class_name, 19 '__module__':module_name, 20 '__signature__':signature, 21 } 22 d.update(obj.__dict__) 23 return d 24 25 obj = json_myobj.MyObj('instance value goes here') 26 print json.dumps(obj, default=convert_to_builtin_type) Sunday, March 20, 2011
  • 44. json 10 def convert_to_builtin_type(obj): 11 """object->dictionary""" 12 print 'convert_to_builtin_type(%r)' % obj 13 class_name = obj.__class__.__name__ 14 module_name = obj.__module__ 15 digest_maker = hmac.new('PyCon2011', 16 module_name + class_name) 17 signature = digest_maker.hexdigest() 18 d = { '__class__':class_name, 19 '__module__':module_name, 20 '__signature__':signature, 21 } 22 d.update(obj.__dict__) 23 return d 24 25 obj = json_myobj.MyObj('instance value goes here') 26 print json.dumps(obj, default=convert_to_builtin_type) Sunday, March 20, 2011
  • 45. json $ python hidden_stdlib/json_dump_default.py convert_to_builtin_type(<MyObj(instance value goes here)>) {"s": "instance value goes here", "__module__": "json_myobj", "__signature__": "426f662f9fe3b3533d9ce7f9dcf8af77", "__class__": "MyObj"} Sunday, March 20, 2011
  • 46. json $ python hidden_stdlib/json_dump_default.py convert_to_builtin_type(<MyObj(instance value goes here)>) {"s": "instance value goes here", "__module__": "json_myobj", "__signature__": "426f662f9fe3b3533d9ce7f9dcf8af77", "__class__": "MyObj"} Sunday, March 20, 2011
  • 47. json $ python hidden_stdlib/json_dump_default.py convert_to_builtin_type(<MyObj(instance value goes here)>) {"s": "instance value goes here", "__module__": "json_myobj", "__signature__": "426f662f9fe3b3533d9ce7f9dcf8af77", "__class__": "MyObj"} Sunday, March 20, 2011
  • 48. json $ python hidden_stdlib/json_dump_default.py convert_to_builtin_type(<MyObj(instance value goes here)>) {"s": "instance value goes here", "__module__": "json_myobj", "__signature__": "426f662f9fe3b3533d9ce7f9dcf8af77", "__class__": "MyObj"} Sunday, March 20, 2011
  • 49. json $ python hidden_stdlib/json_dump_default.py convert_to_builtin_type(<MyObj(instance value goes here)>) {"s": "instance value goes here", "__module__": "json_myobj", "__signature__": "426f662f9fe3b3533d9ce7f9dcf8af77", "__class__": "MyObj"} Sunday, March 20, 2011
  • 50. json 9 def dict_to_object(d): 10 if '__class__' not in d: 11 return d 12 13 class_name = d.pop('__class__') 14 module_name = d.pop('__module__') 15 signature = d.pop('__signature__') 16 17 digest_maker = hmac.new('PyCon2011', 18 module_name + class_name) 19 expected_signature = digest_maker.hexdigest() 20 if signature != expected_signature: 21 raise ValueError('Invalid signature') Sunday, March 20, 2011
  • 51. json 9 def dict_to_object(d): 10 if '__class__' not in d: 11 return d 12 13 class_name = d.pop('__class__') 14 module_name = d.pop('__module__') 15 signature = d.pop('__signature__') 16 17 digest_maker = hmac.new('PyCon2011', 18 module_name + class_name) 19 expected_signature = digest_maker.hexdigest() 20 if signature != expected_signature: 21 raise ValueError('Invalid signature') Sunday, March 20, 2011
  • 52. json 9 def dict_to_object(d): 10 if '__class__' not in d: 11 return d 12 13 class_name = d.pop('__class__') 14 module_name = d.pop('__module__') 15 signature = d.pop('__signature__') 16 17 digest_maker = hmac.new('PyCon2011', 18 module_name + class_name) 19 expected_signature = digest_maker.hexdigest() 20 if signature != expected_signature: 21 raise ValueError('Invalid signature') Sunday, March 20, 2011
  • 53. json 23 print 'Loading "%s" from "%s"' % 24 (class_name, module_name) 25 module = __import__(module_name) 26 class_ = getattr(module, class_name) 27 28 args = dict( (key.encode('ascii'), value) 29 for key, value in d.items()) 30 print 'Instantiating with', args 31 32 inst = class_(**args) 33 return inst Sunday, March 20, 2011
  • 54. json 23 print 'Loading "%s" from "%s"' % 24 (class_name, module_name) 25 module = __import__(module_name) 26 class_ = getattr(module, class_name) 27 28 args = dict( (key.encode('ascii'), value) 29 for key, value in d.items()) 30 print 'Instantiating with', args 31 32 inst = class_(**args) 33 return inst Sunday, March 20, 2011
  • 55. json 23 print 'Loading "%s" from "%s"' % 24 (class_name, module_name) 25 module = __import__(module_name) 26 class_ = getattr(module, class_name) 27 28 args = dict( (key.encode('ascii'), value) 29 for key, value in d.items()) 30 print 'Instantiating with', args 31 32 inst = class_(**args) 33 return inst Sunday, March 20, 2011
  • 56. json 23 print 'Loading "%s" from "%s"' % 24 (class_name, module_name) 25 module = __import__(module_name) 26 class_ = getattr(module, class_name) 27 28 args = dict( (key.encode('ascii'), value) 29 for key, value in d.items()) 30 print 'Instantiating with', args 31 32 inst = class_(**args) 33 return inst Sunday, March 20, 2011
  • 57. json 35 for encoded_object in [ 36 ''' 37 [{"s": "instance value goes here", 38 "__signature__": "426f662f9fe3b3533d9ce7f9dcf8af77", 39 "__module__": "json_myobj", "__class__": "MyObj"}] 40 ''', 41 42 # careful! 43 ''' 44 [{"path": "/etc/passwd", 45 "__signature__": "426f662f9fe3b3533d9ce7f9dcf8af77", 46 "__module__": "os", "__class__": "unlink"}] 47 ''', 48 ]: Sunday, March 20, 2011
  • 58. json 35 for encoded_object in [ 36 ''' 37 [{"s": "instance value goes here", 38 "__signature__": "426f662f9fe3b3533d9ce7f9dcf8af77", 39 "__module__": "json_myobj", "__class__": "MyObj"}] 40 ''', 41 42 # careful! 43 ''' 44 [{"path": "/etc/passwd", 45 "__signature__": "426f662f9fe3b3533d9ce7f9dcf8af77", 46 "__module__": "os", "__class__": "unlink"}] 47 ''', 48 ]: Sunday, March 20, 2011
  • 59. json 49 try: 50 myobj_instance = json.loads( 51 encoded_object, 52 object_hook=dict_to_object, 53 ) 54 print myobj_instance 55 except Exception as err: 56 print 'ERROR:', err 57 print Sunday, March 20, 2011
  • 60. json $ python hidden_stdlib/json_load_object_hook.py Loading "MyObj" from "json_myobj" Instantiating with {'s': u'instance value goes here'} [<MyObj(instance value goes here)>] ERROR: Invalid signature Sunday, March 20, 2011
  • 61. json $ python hidden_stdlib/json_load_object_hook.py Loading "MyObj" from "json_myobj" Instantiating with {'s': u'instance value goes here'} [<MyObj(instance value goes here)>] ERROR: Invalid signature Sunday, March 20, 2011
  • 62. Error Handling sys.excepthook Sunday, March 20, 2011
  • 63. sys.excepthook 6 def main(): 7 # do some work 8 raise RuntimeError('Helpful error message') 9 10 if __name__ == '__main__': 11 main() $ python hidden_stdlib/sys_excepthook_no_handler.py Traceback (most recent call last): File ".../hidden_stdlib/sys_excepthook_no_handler.py", line 11, in <module> main() File ".../hidden_stdlib/sys_excepthook_no_handler.py", line 8, in main raise RuntimeError('Helpful error message') RuntimeError: Helpful error message Sunday, March 20, 2011
  • 64. sys.excepthook 6 import sys 7 8 def main(): 9 try: 10 # do some work 11 raise RuntimeError('Helpful error message') 12 except Exception as err: 13 sys.stderr.write('ERROR: %sn' % err) 14 15 if __name__ == '__main__': 16 main() $ python hidden_stdlib/sys_excepthook_big_block.py ERROR: Helpful error message Sunday, March 20, 2011
  • 65. sys.excepthook 6 import sys 7 8 def quiet_errors(exc_type, exc_value, traceback): 9 sys.stderr.write('ERROR: %sn' % exc_value) 10 11 sys.excepthook = quiet_errors 12 13 def main(): 14 # do some work 15 raise RuntimeError('Helpful error message') 16 17 if __name__ == '__main__': 18 main() $ python hidden_stdlib/sys_excepthook.py ERROR: Helpful error message Sunday, March 20, 2011
  • 66. sys.excepthook 6 import sys 7 8 def quiet_errors(exc_type, exc_value, traceback): 9 sys.stderr.write('ERROR: %sn' % exc_value) 10 11 sys.excepthook = quiet_errors 12 13 def main(): 14 # do some work 15 raise RuntimeError('Helpful error message') 16 17 if __name__ == '__main__': 18 main() $ python hidden_stdlib/sys_excepthook.py ERROR: Helpful error message Sunday, March 20, 2011
  • 67. Communicating logging with Users Sunday, March 20, 2011
  • 68. logging loggers Sunday, March 20, 2011
  • 69. logging loggers Sunday, March 20, 2011
  • 70. logging 10 # Log verbosely 11 root_logger = logging.getLogger('') 12 root_logger.setLevel(logging.DEBUG) 13 14 # Set up console output to stderr 15 console = logging.StreamHandler(sys.stderr) 16 console_format = '%(message)s' 17 console.setFormatter(logging.Formatter(console_format)) 18 console.setLevel(logging.INFO) # TODO: command line switch 19 root_logger.addHandler(console) Sunday, March 20, 2011
  • 71. logging 10 # Log verbosely 11 root_logger = logging.getLogger('') 12 root_logger.setLevel(logging.DEBUG) 13 14 # Set up console output to stderr 15 console = logging.StreamHandler(sys.stderr) 16 console_format = '%(message)s' 17 console.setFormatter(logging.Formatter(console_format)) 18 console.setLevel(logging.INFO) # TODO: command line switch 19 root_logger.addHandler(console) Sunday, March 20, 2011
  • 72. logging 10 # Log verbosely 11 root_logger = logging.getLogger('') 12 root_logger.setLevel(logging.DEBUG) 13 14 # Set up console output to stderr 15 console = logging.StreamHandler(sys.stderr) 16 console_format = '%(message)s' 17 console.setFormatter(logging.Formatter(console_format)) 18 console.setLevel(logging.INFO) # TODO: command line switch 19 root_logger.addHandler(console) Sunday, March 20, 2011
  • 73. logging 10 # Log verbosely 11 root_logger = logging.getLogger('') 12 root_logger.setLevel(logging.DEBUG) 13 14 # Set up console output to stderr 15 console = logging.StreamHandler(sys.stderr) 16 console_format = '%(message)s' 17 console.setFormatter(logging.Formatter(console_format)) 18 console.setLevel(logging.INFO) # TODO: command line switch 19 root_logger.addHandler(console) Sunday, March 20, 2011
  • 74. logging 10 # Log verbosely 11 root_logger = logging.getLogger('') 12 root_logger.setLevel(logging.DEBUG) 13 14 # Set up console output to stderr 15 console = logging.StreamHandler(sys.stderr) 16 console_format = '%(message)s' 17 console.setFormatter(logging.Formatter(console_format)) 18 console.setLevel(logging.INFO) # TODO: command line switch 19 root_logger.addHandler(console) Sunday, March 20, 2011
  • 75. logging 21 # Include debug messages when logging to a file 22 file_handler = logging.handlers.RotatingFileHandler( 23 'logging_example.log', # use a full path 24 ) 25 file_format = '%(asctime)s %(levelname)6s %(name)s % (message)s' 26 file_handler.setFormatter(logging.Formatter(file_format)) 27 file_handler.setLevel(logging.DEBUG) 28 root_logger.addHandler(file_handler) Sunday, March 20, 2011
  • 76. logging 21 # Include debug messages when logging to a file 22 file_handler = logging.handlers.RotatingFileHandler( 23 'logging_example.log', # use a full path 24 ) 25 file_format = '%(asctime)s %(levelname)6s %(name)s % (message)s' 26 file_handler.setFormatter(logging.Formatter(file_format)) 27 file_handler.setLevel(logging.DEBUG) 28 root_logger.addHandler(file_handler) Sunday, March 20, 2011
  • 77. logging 21 # Include debug messages when logging to a file 22 file_handler = logging.handlers.RotatingFileHandler( 23 'logging_example.log', # use a full path 24 ) 25 file_format = '%(asctime)s %(levelname)6s %(name)s % (message)s' 26 file_handler.setFormatter(logging.Formatter(file_format)) 27 file_handler.setLevel(logging.DEBUG) 28 root_logger.addHandler(file_handler) Sunday, March 20, 2011
  • 78. logging 30 # Log sample messages with different levels 31 log = logging.getLogger(__name__) 32 log.info('on the console and in the file') 33 log.debug('only in the file') 34 log.error('simple error message') 35 36 # Replace excepthook with logger 37 def log_exception(exc_type, exc_value, traceback): 38 logging.getLogger(__name__).error(exc_value) 39 sys.excepthook = log_exception 40 41 # Send exceptions to the logger automatically 42 raise RuntimeError('failure message') Sunday, March 20, 2011
  • 79. logging 30 # Log sample messages with different levels 31 log = logging.getLogger(__name__) 32 log.info('on the console and in the file') 33 log.debug('only in the file') 34 log.error('simple error message') 35 36 # Replace excepthook with logger 37 def log_exception(exc_type, exc_value, traceback): 38 logging.getLogger(__name__).error(exc_value) 39 sys.excepthook = log_exception 40 41 # Send exceptions to the logger automatically 42 raise RuntimeError('failure message') Sunday, March 20, 2011
  • 80. logging 30 # Log sample messages with different levels 31 log = logging.getLogger(__name__) 32 log.info('on the console and in the file') 33 log.debug('only in the file') 34 log.error('simple error message') 35 36 # Replace excepthook with logger 37 def log_exception(exc_type, exc_value, traceback): 38 logging.getLogger(__name__).error(exc_value) 39 sys.excepthook = log_exception 40 41 # Send exceptions to the logger automatically 42 raise RuntimeError('failure message') Sunday, March 20, 2011
  • 81. logging $ python hidden_stdlib/logging_example.py on the console and in the file simple error message failure message Sunday, March 20, 2011
  • 82. logging $ python hidden_stdlib/logging_example.py on the console and in the file simple error message failure message $ cat logging_example.log 2011-02-13 11:28:30,036 INFO __main__ on the console and in the file 2011-02-13 11:28:30,036 DEBUG __main__ only in the file 2011-02-13 11:28:30,036 ERROR __main__ simple error message 2011-02-13 11:28:30,036 ERROR __main__ failure message Sunday, March 20, 2011
  • 83. logging $ python hidden_stdlib/logging_example.py on the console and in the file simple error message failure message $ cat logging_example.log 2011-02-13 11:28:30,036 INFO __main__ on the console and in the file 2011-02-13 11:28:30,036 DEBUG __main__ only in the file 2011-02-13 11:28:30,036 ERROR __main__ simple error message 2011-02-13 11:28:30,036 ERROR __main__ failure message Sunday, March 20, 2011
  • 84. logging $ python hidden_stdlib/logging_example.py on the console and in the file simple error message failure message $ cat logging_example.log 2011-02-13 11:28:30,036 INFO __main__ on the console and in the file 2011-02-13 11:28:30,036 DEBUG __main__ only in the file 2011-02-13 11:28:30,036 ERROR __main__ simple error message 2011-02-13 11:28:30,036 ERROR __main__ failure message Sunday, March 20, 2011
  • 85. logging $ python hidden_stdlib/logging_example.py on the console and in the file simple error message failure message $ cat logging_example.log 2011-02-13 11:28:30,036 INFO __main__ on the console and in the file 2011-02-13 11:28:30,036 DEBUG __main__ only in the file 2011-02-13 11:28:30,036 ERROR __main__ simple error message 2011-02-13 11:28:30,036 ERROR __main__ failure message Sunday, March 20, 2011
  • 86. logging Traceback Handler Sunday, March 20, 2011
  • 87. logging 36 # Tracebacks should only go to the file 37 traceback_log = logging.getLogger('traceback') 38 traceback_log.propagate = False 39 traceback_log.setLevel(logging.ERROR) 40 traceback_log.addHandler(file_handler) 41 42 # Replace excepthook with logger 43 def log_exception(exc_type, exc_value, traceback): 44 logging.getLogger(__name__).error(exc_value) 45 logging.getLogger('traceback').error( 46 exc_value, 47 exc_info=(exc_type, exc_value, traceback), 48 ) 49 sys.excepthook = log_exception 50 51 # Send exceptions to the logger automatically 52 raise RuntimeError('failure message') Sunday, March 20, 2011
  • 88. logging 36 # Tracebacks should only go to the file 37 traceback_log = logging.getLogger('traceback') 38 traceback_log.propagate = False 39 traceback_log.setLevel(logging.ERROR) 40 traceback_log.addHandler(file_handler) 41 42 # Replace excepthook with logger 43 def log_exception(exc_type, exc_value, traceback): 44 logging.getLogger(__name__).error(exc_value) 45 logging.getLogger('traceback').error( 46 exc_value, 47 exc_info=(exc_type, exc_value, traceback), 48 ) 49 sys.excepthook = log_exception 50 51 # Send exceptions to the logger automatically 52 raise RuntimeError('failure message') Sunday, March 20, 2011
  • 89. logging $ python hidden_stdlib/logging_example_tracebacks.py on the console and in the file simple error message failure message $ cat logging_example.log 2011-02-13 11:33:22,592 INFO __main__ on the console and in the file 2011-02-13 11:33:22,592 DEBUG __main__ only in the file 2011-02-13 11:33:22,592 ERROR __main__ simple error message 2011-02-13 11:33:22,592 ERROR __main__ failure message 2011-02-13 11:33:22,593 ERROR traceback failure message Traceback (most recent call last): File "hidden_stdlib/logging_example_tracebacks.py", line 52, in <module> raise RuntimeError('failure message') RuntimeError: failure message Sunday, March 20, 2011
  • 90. More Information • http://docs.python.org/ • http://www.doughellmann.com/PyMOTW/ • https://bitbucket.org/dhellmann/hidden_stdlib/ • The Python Standard Library By Example, Doug Hellmann • Python Essential Reference, David Beazley Sunday, March 20, 2011
  • 91. Image Credits 2. http://www.uottawa.ca.libguides.com/content.php?pid=14825&sid=117658 4. http://www.flickr.com/photos/seditiouscanary/1279041211/ 5. http://www.flickr.com/photos/elrentaplats/4902419885/in/photostream/ 6. http://www.flickr.com/photos/elrentaplats/4903005444/in/photostream/ 18. http://etc.usf.edu/clipart/58000/58063/58063_column_types.htm 34. http://www.flickr.com/photos/akeg/3077866744/ 44. http://www.flickr.com/photos/yum9me/2807475045/ 64. http://commons.wikimedia.org/wiki/File:Zeichen_trissturis_error101.png 72. http://www.flickr.com/photos/myklroventine/3545127104/in/ faves-40068198@N04/ Sunday, March 20, 2011