Slides for the Cluj.py meetup where we explored the inner workings of CPython, the reference implementation of Python. Includes examples of writing a C extension to Python, and introduces Cython - ultimately the sanest way of writing C extensions. Also check out the code samples on GitHub: https://github.com/trustyou/meetups/tree/master/python-c

1. Extending Python in C
Cluj.py meetup, Nov 19th
Steffen Wenz, CTO TrustYou

2. Goals of today’s talk
● Look behind the scenes of the CPython interpreter -
gain insights into “how Python works”
● Explore the CPython C API
● Build a Python extension in C
● Introduction to Cython

3. Who are we?
● For each hotel on the
planet, provide a
summary of all reviews
● Expertise:
○ NLP
○ Machine Learning
○ Big Data
● Clients: …

5. TrustYou Tech Stack
Batch Layer
● Hadoop (HDP 2.1)
● Python
● Pig
● Luigi
Service Layer
● PostgreSQL
● MongoDB
● Redis
● Cassandra
Data Data Queries
Hadoop cluster (100 nodes) Application machines

6. Let’s dive in! Assigning an integer
a = 4 PyObject* a =
PyInt_FromLong(4);
// what's the
difference to
int a = 4?
Documentation: PyInt_FromLong

7. List item access
x = xs[i] PyObject* x =
PyList_GetItem(xs,
i);
Documentation: PyList_GetItem

8. Returning None …
return None Py_INCREF(Py_None);
return Py_None;
Documentation: Py_INCREF

9. Calling a function
foo(1337, "bar") // argument list
PyObject *args = Py_BuildValue
("is", 1337, "bar");
// make call
PyObject_CallObject(foo,
args);
// release arguments
Py_DECREF(args);
Documentation: Py_BuildValue,
PyObject_CallObject

10. What’s the CPython C API?
● API to manipulate Python objects, and interact with
Python code, from C/C++
● Purpose: Extend Python with new modules/types
● Why?

11. CPython internals
def slangify(s):
return s + ", yo!"
C API
Compiler Interpreter
>>> slangify("hey")
'hey, yo!'
|x00x00dx01x00x17S
Not true for Jython, IronPython, PyPy, Stackless …

12. Why is Python slow?
a = 1
a = a + 1
int a = 1;
a++;

13. Why is Python slow?
class Point:
def __init__(self, x, y):
self.x = x; self.y = y
p = Point(1, 2)
print p.x
typedef struct { int x, y; }
point;
int main() {
point p = {1, 2};
printf("%i", p.x);
}

14. Why is Python slow?
The GIL

15. Writing in C
● No OOP :
typedef struct { /* ... */ } complexType;
void fun(complexType* obj, int x, char* y) {
// ...
}
● Macros for code generation:
#define SWAP(x,y) {int tmp = x; x = y; y = tmp;}
SWAP(a, b);

16. Writing in C
● Manual memory management:
○ C: static, stack, malloc/free
○ Python C API: Reference counting
● No exceptions
○ Error handling via returning values
○ CPython: return null; signals an error

17. Reference Counting
void Py_INCREF(PyObject *o)
Increment the reference count for object o. The object must not be NULL; if you aren’t sure that it isn’t NULL, use
Py_XINCREF().
= I want to hold on to this object and use it again after a while*
*) any interaction with Python interpreter that may invalidate my reference
void Py_DECREF(PyObject *o)
Decrement the reference count for object o. The object must not be NULL; if you aren’t sure that it isn’t NULL, use
Py_XDECREF(). If the reference count reaches zero, the object’s type’s deallocation function (which must not
beNULL) is invoked.
= I’m done, and don’t care if the object is discarded at this call
See documentation

18. Anatomy of a refcount bug
void buggy(PyObject *list)
{
PyObject *item = PyList_GetItem(list, 0); // borrowed ref.
PyList_SetItem(list, 1, PyInt_FromLong(0L)); // calls
destructor of previous element
PyObject_Print(item, stdout, 0); // BUG!
}

19. Our First Extension Module

20. Adding integers in C
>>> import arithmetic
>>> arithmetic.add(1, 1337)
1338

21. #include <Python.h>
static PyObject*
arithmetic_add(PyObject* self, PyObject* args)
{
int i, j;
PyArg_ParseTuple(args, "ii", &i, &j);
PyObject* sum = PyInt_FromLong(i + j);
return sum;
}

22. static PyObject*
arithmetic_add(PyObject* self, PyObject* args)
{
int i, j;
PyObject* sum = NULL;
if (!PyArg_ParseTuple(args, "ii", &i, &j))
goto error;
sum = PyInt_FromLong(i + j);
if (sum == NULL)
goto error;
return sum;
error:
Py_XDECREF(sum);
return NULL;
}

23. Boilerplate²
static PyMethodDef ArithmeticMethods[] = {
{"add", arithmetic_add, METH_VARARGS, "Add two integers."},
{NULL, NULL, 0, NULL} // sentinel
};
PyMODINIT_FUNC
initarithmetic(void)
{
(void) Py_InitModule("arithmetic", ArithmeticMethods);
}

24. … and build your module
from distutils.core import setup, Extension
module = Extension("arithmetic", sources=["arithmeticmodule.c"])
setup(
name="Arithmetic",
version="1.0",
ext_modules=[module]
)

25. $ sudo python setup.py install
# build with gcc, any compiler errors & warnings are shown here
$ python
>>> import arithmetic
>>> arithmetic
<module 'arithmetic' from '/usr/local/lib/python2.7/dist-packages/
arithmetic.so'>
>>> arithmetic.add
<built-in function add>
>>> arithmetic.add(1, "1337")
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: an integer is required

26. Why on earth
would I do that?

27. Why go through all this trouble?
● Performance
○ C extensions & Cython optimize CPU-bound code
(vs. memory-bound, IO-bound)
○ Pareto principle: 20% of the code responsible for
80% of the runtime
● Also: Interfacing with existing C/C++ code

28. Is my Python code performance-critical?
import cProfile, pstats, sys
pr = cProfile.Profile()
pr.enable()
setup()
# run code you want to profile
pr.disable()
stats = pstats.Stats(pr, stream=sys.stdout).sort_stats("time")
stats.print_stats()

29. 55705589 function calls (55688041 primitive calls) in 69.216 seconds
Ordered by: internal time
ncalls tottime percall cumtime percall filename:lineno(function)
45413 21.856 0.000 21.856 0.000 {method 'get' of 'pytc.HDB' objects}
32275 9.490 0.000 9.656 0.000 /usr/local/lib/python2.7/dist-packages/simplejson/decoder.py:376(raw_decode)
18760 6.403 0.000 12.797 0.001 /home/steffen/apps/group/lib/util/timeseries.py:29(reindex_pad)
56992 2.586 0.000 2.624 0.000 {sorted}
1383832 2.244 0.000 2.244 0.000 /home/steffen/apps/group/lib/hotel/index.py:231(<lambda>)
2708692 1.845 0.000 5.657 0.000 /home/steffen/apps/group/lib/hotel/index.py:21(<genexpr>)
497989 1.718 0.000 2.456 0.000 {_heapq.heapreplace}
4734466 1.624 0.000 2.491 0.000 /home/steffen/apps/group/lib/util/timeseries.py:43(<genexpr>)
346738 1.475 0.000 1.475 0.000 /usr/lib/python2.7/json/decoder.py:371(raw_decode)
510726 1.354 0.000 10.432 0.000 /usr/lib/python2.7/heapq.py:357(merge)
2691966 1.310 0.000 1.310 0.000 /home/steffen/apps/group/lib/util/timeseries.py:21(float_parse)
357260 1.160 0.000 5.122 0.000 /home/steffen/apps/group/lib/hotel/index.py:471(<genexpr>)
5348564 0.912 0.000 0.912 0.000 /home/steffen/apps/group/lib/util/timeseries.py:90(<genexpr>)
758026 0.882 0.000 0.882 0.000 {method 'match' of '_sre.SRE_Pattern' objects}
9470443 0.868 0.000 0.868 0.000 {method 'append' of 'list' objects}
4715746 0.867 0.000 0.867 0.000 /home/steffen/apps/group/lib/util/timeseries.py:31(bound)
1 0.857 0.857 69.220 69.220 /home/steffen/apps/group/lib/pages/table_page.py:37(calculate)
644766 0.839 0.000 1.752 0.000 {sum}

30. You can’t observe without changing …
import timeit
def setup():
pass
def stmt():
pass
print timeit.timeit(stmt=stmt, setup=setup, number=100)

31. Example: QuickSort

32. Pythonic QuickSort
def quicksort(xs):
if len(xs) <= 1:
return xs
middle = len(xs) / 2
pivot = xs[middle]
del xs[middle]
left, right = [], []
for x in xs:
append_to = left if x < pivot else right
append_to.append(x)
return quicksort(left) + [pivot] + quicksort(right)

33. Results: Python vs. C extension
Pythonic QuickSort: 2.0s
C extension module: 0.092s

34. Cython

35. Adding integers in Cython
# add.pyx
def add(i, j):
return i + j
# main.py
import pyximport; pyximport.install()
import add
if __name__ == "__main__":
print add.add(1, 1337)

36. What is Cython?
● Compiles Python to C code
● “Superset” of Python: Accepts type annotations to
compile more efficient code (optional!)
cdef int i = 2
● No reference counting, error handling, boilerplate …
plus nicer compiling workflows

37. Results:
Pythonic QuickSort: 2.0s
C extension module: 0.092s
Cython QuickSort (unchanged): 0.82s

38. cdef partition(xs, int left, int right, int pivot_index):
cdef int pivot = xs[pivot_index]
cdef int el
xs[pivot_index], xs[right] = xs[right], xs[pivot_index]
pivot_index = left
for i in xrange(left, right):
el = xs[i]
if el <= pivot:
xs[i], xs[pivot_index] = xs[pivot_index], xs[i]
pivot_index += 1
xs[pivot_index], xs[right] = xs[right], xs[pivot_index]
return pivot_index
def quicksort(xs, left=0, right=None):
if right is None:
right = len(xs) - 1
if left < right:
middle = (left + right) / 2
pivot_index = partition(xs, left, right, middle)
quicksort(xs, left, pivot_index - 1)
quicksort(xs, pivot_index + 1, right)

39. Results:
Pythonic QuickSort: 2.0s
C extension module: 0.092s
Cython QuickSort (unchanged): 0.82s
Cython QuickSort (C-like): 0.37s
● Unscientific result. Cython can be faster than hand-written
C extensions!

40. Further Reading on Cython
See code samples on ● O’Reilly Book
TrustYou GitHub account:
https://github.
com/trustyou/meetups/tre
e/master/python-c

41. TrustYou wants you!
We offer positions
in Cluj & Munich:
● Data engineer
● Application developer
● Crawling engineer
Write me at swenz@trustyou.net, check out our website,
or see you at the next meetup!

42. Thank you!

43. Python Bytecode
>>> def slangify(s):
... return s + ", yo!"
...
>>> slangify.func_code.co_code
'|x00x00dx01x00x17S'
>>> import dis
>>> dis.dis(slangify)
2 0 LOAD_FAST 0 (s)
3 LOAD_CONST 1 (', yo!')
6 BINARY_ADD
7 RETURN_VALUE

44. Anatomy of a memory leak
void buggier()
{
PyObject *lst = PyList_New(10);
return Py_BuildValue("Oi", lst, 10); // increments refcount
}
// read the doc carefully before using *any* C API function