15. Extending Python in C/C++

♦ C/C++ registers functions to Python

♦ For optimization and integration

Extending topics

♦ Integration overview

♦ C extension modules

♦ C extension types

♦ Wrapper classes

♦ SWIG glue code generator

Python in-process integration model

► Extending: optimization, legacy code reuse, tasks outside Python’s domain

► Embedding: customization, event dispatch

► Combined modes: GUI extending calls, events via embedding

► Extending tools: SWIG, SIP, Boost.Python, Cython, Ctypes (2.5+)

Review: Python tool-set layers

♦ Built-ins

● Lists, dictionaries, strings, library modules, etc.

● High-level tools for simple, fast programming

♦ Python extensions

● Functions, classes, modules

● For adding extra features, and new object types

♦ C extensions and embedding ← you are here

● C modules, C types, runtime API

● For integrating external systems, optimizing components, customization

Why integration?

♦ Optimization

● time-critical components

♦ Customization

● on-site/end-user changes

♦ Component reuse

● wrapping C libraries in a Python front-end

“Hybrid development”

● Python is optimized for speed-of-development

● Python is designed for multi-language systems

● Leverages both development and execution speed

Integration modes

♦ Extending

● Python → C/C++

● For optimization, library integration, special tasks

● Via registering C function pointers

♦ Embedding

● C/C++ → Python

● For system customizations, callbacks

● Via calling Python API functions

♦ May be mixed arbitrarily (Python libs are reentrant)

● Python → C/C++ → Python → C/C++ . . .

Extending basics

● Extending requires glue code between Python and C library for data translation

● Glue code can be written in C, or generated in C with tools like SWIG

● Glue code can also be written in Python with ctypes, std lib module in Python 2.5+

Extending Tools

■ SWIG: generate glue code from C/C++ declarations

■ SIP: a smaller SWIG

■ Ctypes: write glue code in Python (std lib in 2.5)

■ Boost.Python: write glue with C++ template code

■ Cython (was PyRex): a Python/C combo for new extensions

■ CXX: Python API for C++

■ F2py, pyfort: glue code for Fortran

A simple C extension module

♦ Build a C extension module for Python, called “environ”

♦ Wrap (export) the C library’s ‘getenv’ function for use in Python code

file: environ.c

#include <Python.h>

#include <stdlib.h>

/* Functions */

static PyObject * /* returns object */

wrap_getenv(PyObject *self, PyObject *args)

{ /* args from python */

char *varName, *varValue;

PyObject *returnObj = NULL; /* null=exception */

if (PyArg_Parse(args, "s", &varName)) { /*P->C*/

varValue = getenv(varName);

if (varValue != NULL)

returnObj = Py_BuildValue("s", varValue); /*C->P*/

else

PyErr_SetString(PyExc_SystemError, "bad getenv");

}

return returnObj;

}

/* Registration */

static struct PyMethodDef environ_methods[] = {

{"getenv", wrap_getenv}, /* name, address */

{NULL, NULL}

};

/* Initialization */

void initenviron() /* called by Python */

{ /* on first import */

(void) Py_InitModule("environ", environ_methods);

}

Using the C module in Python

♦ Used just like Python modules

♦ Serve same roles: name-space

♦ C functions handle all data conversions

♦ C raises exceptions by returning ‘NULL’

% python

>>> import environ

>>> environ.getenv("USER")

'mlutz'

>>> environ.getenv("PYTHONPATH") # yes, it’s worked that long!

'.:/opt/local/src/Python-1.4/Lib'

>>> dir(environ)

['__doc__', '__file__', '__name__', 'getenv']

>>> environ.__file__

'./environ.so'

C module structure

♦ API tools

● Python API symbols start with a ‘Py’ prefix

● ‘PyArg_Parse’ converts arguments to C

● ‘Py_BuildValue’ C data to Python return object

● ‘PyObject*’ for generic Python objects in C

♦ Module method functions

● ‘self’ argument (not used for modules)

● ‘args’—Python tuple containing passed arguments

♦ Name-to-address registration table

● Maps name strings to C function address

● Null table terminator

♦ Module initialization function

● Called by Python when the module is first imported

● ‘Py_InitModule’ initializes module attributes dictionary

● ‘initenviron’ called by name (non-static)

Binding C extensions to Python

♦ Static binding… rebuild Python

♦ Dynamic binding… load when imported

Static binding

1. Add file to Python tree

Put (or link to) the source or object file in Modules directory of Python source tree

2. Add line to Modules/Setup

Add a line to the Modules/Setup file in the Python source tree: "environ environ.c" (config table)

3. Rebuild Python

Rerun the ‘make’ command at the top-level of Python’s source tree directory

Dynamic Binding

♦ Loaded into process on first import

♦ Doesn’t require access to Python source

♦ Doesn’t require rebuilding Python

♦ Makes for smaller executable/process

♦ Compile/link details are platform-specific

1. Compile into shareable

Compile C module into a sharable (dynamic-load) object file: .so, .sl, .dll, etc. (see makefiles below and in embedding unit)

2. Add to module search path

Put the shareable object file in a directory named on the ‘$PYTHONPATH’ environment variable

Makefile example, dynamic binding on Linux

File makefile.environ

##########################################################

# Compile environ.c into a shareable object file on

# Linux, to be loaded dynamically when first imported,

# whether from interactive, stand-alone, or embedded code.

# To use, type make -f makefile.environ, at your shell;

# to run, make sure environ.so is on a dir on PYTHONPATH.

# To link statically with Python instead, add line like:

# environ ~/examples/Part1/Preview/Integrate/environ.c

# to Modules/Setup (or add a link to environ.c in Modules,

# and add a line like environ environ.c) and re-make

# python itself; this works on any platform, and no extra

# makefile like this one is needed;

# To make a shareable on Solaris, you might instead say:

# cc xxx.c -c -KPIC -o xxx.o

# ld -G xxx.o -o xxx.so

# rm xxx.o

# On other platforms, it's more different still; see

# your c or c++ compiler's documentation for details

##########################################################

PY = /home/mark/python1.5.2-ddjcd/Python-1.5.2

environ.so: environ.c

gcc environ.c -g -I$(PY)/Include -I$(PY) -fpic -shared -o environ.so

clean:

rm -f environ.so

#or-- environ.so: environ.c

# gcc environ.c -c -g -fpic -I$(PY)/Include -I$(PY) -o environ.o

# gcc -shared environ.o -o environ.so

# rm -f environ.o

Data conversions: Python →← C

♦ ‘PyArg_Parse’ converts arguments to C

♦ ‘Py_BuildValue’ C data to Python return object

♦ ‘PyArg_ParseTuple’ assumes it’s converting a tuple

♦ Other API tools handle type-specific conversions

Common conversion codes

Format code	C data-type	Python object-type
“s”	char*	string
“s#”	char*, int	string, length
“i”	int	integer
“l”	long int	integer
“c”	char (or int for build)	string
“f”	float	floating-point
“d”	double	floating-point
“O”	PyObject*	a raw object
“(items)”	targets or values	nested tuple
“[items]”	series of arg/value	list
“{items}”	series of “key,value”	dictionary

♦ Warning:

● “s” returns pointer to string in Python: use it or lose it

C extension types

♦ Creation of multiple instances

Each is a new C ‘struct’, not a dictionary

♦ Overloading operators and type operations

Via type-descriptor tables, not method names

♦ Types now also support inheritance like classes

Type file components

1. A C ‘struct’ used to hold per-instance data

2. Instance method functions and registration table

3. Functions to handle general type operations

4. Functions to handle specific type category operations

5. Type-descriptor tables: register operation handlers

6. A C extension module: exports instance constructor

A ‘skeleton’ C extension type

♦ A C string-stack type

♦ Implements push/pop methods

♦ Overloads sequence and type operators

file: stacktyp.c

/****************************************************

* stacktyp.c: a character-string stack data-type;

* a C extension type, for use in Python programs;

* stacktype module clients can make multiple stacks;

* similar to stackmod, but 'self' is the instance,

* and we can overload sequence operators here;

****************************************************/

#include "Python.h"

static PyObject *ErrorObject; /* local exception */

#define onError(message) \

{ PyErr_SetString(ErrorObject, message); return NULL; }

/**********************************************************

* STACK-TYPE INFORMATION

**********************************************************/

#define MAXCHARS 2048

#define MAXSTACK MAXCHARS

typedef struct { /* stack instance object */

PyObject_HEAD /* python header */

int top, len; /* + per-instance info */

char *stack[MAXSTACK];

char strings[MAXCHARS];

} stackobject;

staticforward PyTypeObject Stacktype; /* type descriptor */

#define is_stackobject(v) ((v)->ob_type == &Stacktype)

/**********************************************************

* INSTANCE METHODS

**********************************************************/

static PyObject * /* on "instance.push(arg)" */

stack_push(self, args) /* 'self' is the instance */

stackobject *self; /* 'args' passed to method */

PyObject *args;

{

char *pstr;

if (!PyArg_ParseTuple(args, "s", &pstr))

return NULL;

[. . .]

Py_INCREF(Py_None);

return Py_None;

}

static PyObject *

stack_pop(self, args)

stackobject *self;

PyObject *args; /* on "instance.pop()" */

{ [. . .]

return Py_BuildValue("s", "not implemented");

}

static PyObject *

stack_top(self, args)

stackobject *self;

PyObject *args;

{ [. . .]

}

static PyObject *

stack_empty(self, args)

stackobject *self;

PyObject *args;

{ [. . .]

}

/* instance methods */

static struct PyMethodDef stack_methods[] = {

{"push", stack_push, 1}, /* name, addr */

{"pop", stack_pop, 1},

{"top", stack_top, 1},

{"empty", stack_empty, 1},

{NULL, NULL}

};

/**********************************************************

* BASIC TYPE-OPERATIONS

**********************************************************/

static stackobject * /* on "x = stacktype.Stack()" */

newstackobject() /* instance constructor */

{

stackobject *self;

self = PyObject_NEW(stackobject, &Stacktype);

if (self == NULL)

return NULL; /* raise exception */

self->top = 0;

self->len = 0;

return self; /* a new type-instance */

}

static void /* instance destructor */

stack_dealloc(self) /* frees instance struct */

stackobject *self;

{

PyMem_DEL(self);

}

static int

stack_print(self, fp, flags)

stackobject *self;

FILE *fp;

int flags; /* print self to file */

{ [. . .]

}

static PyObject *

stack_getattr(self, name) /* on "instance.attr" */

stackobject *self; /* bound-method or member */

char *name;

{

if (strcmp(name, "len") == 0)

return Py_BuildValue("i", self->len);

return

Py_FindMethod(stack_methods, (PyObject *)self, name);

}

static int

stack_compare(v, w)

stackobject *v, *w; /* return -1, 0 or 1 */

{ [. . .]

}

/**********************************************************

* SEQUENCE TYPE-OPERATIONS

**********************************************************/

static int

stack_length(self)

stackobject *self; /* on "len(instance)" */

{

[. . .]

}

static PyObject *

stack_concat(self, other)

stackobject *self; /* on "instance + other" */

PyObject *other;

{ [. . .] /* return new stack instance */

}

static PyObject *

stack_repeat(self, n) /* on "instance * N" */

stackobject *self;

int n;

{ [. . .]

}

static PyObject *

stack_item(self, index) /* on x[i], for, in */

stackobject *self;

int index;

{

if (index < 0 || index >= self->top) {

PyErr_SetString(PyExc_IndexError, "out-of-bounds");

return NULL;

}

else

return Py_BuildValue("s", self->stack[index]);

}

static PyObject *

stack_slice(self, ilow, ihigh)

stackobject *self;

int ilow, ihigh;

{

/* return ilow..ihigh slice of self--new object */

onError("slicing not yet implemented")

}

/**********************************************************

* TYPE DESCRIPTORS: MORE REGISTRATION

**********************************************************/

static PySequenceMethods stack_as_sequence = {

(inquiry) stack_length,

(binaryfunc) stack_concat,

(intargfunc) stack_repeat,

(intargfunc) stack_item,

(intintargfunc) stack_slice,

(intobjargproc) 0, /* setitem */

(intintobjargproc) 0, /* setslice */

};

static PyTypeObject Stacktype = { /* type descriptor */

/* type header */

PyObject_HEAD_INIT(&PyType_Type)

0, /* ob_size */

"stack", /* name */

sizeof(stackobject), /* basicsize */

0, /* itemsize */

/* standard methods */

(destructor) stack_dealloc, /* dealloc */

(printfunc) stack_print, /* print */

(getattrfunc) stack_getattr, /* getattr */

(setattrfunc) 0, /* setattr */

(cmpfunc) stack_compare, /* compare */

(reprfunc) 0, /* repr */

/* type categories */

0, /* number ops */

&stack_as_sequence, /* sequence ops */

0, /* mapping ops */

/* more methods */

(hashfunc) 0, /* "dict[x]" */

(binaryfunc) 0, /* "x()" */

(reprfunc) 0, /* "str(x)" */

}; /* plus others: see Include/object.h */

/*********************************************************

* MODULE LOGIC: CONSTRUCTOR FUNCTION

**********************************************************/

static PyObject *

stacktype_new(self, args) /* on "x = stacktype.Stack()" */

PyObject *self;

PyObject *args;

{

if (!PyArg_ParseTuple(args, "")) /* Module func */

return NULL;

return (PyObject *)newstackobject();

}

static struct PyMethodDef stacktype_methods[] = {

{"Stack", stacktype_new, 1},

{NULL, NULL}

};

void

initstacktype() /* on first "import stacktype" */

{

PyObject *m, *d;

m = Py_InitModule("stacktype", stacktype_methods);

d = PyModule_GetDict(m);

ErrorObject = Py_BuildValue("s", "stacktype.error");

PyDict_SetItemString(d, "error", ErrorObject);

if (PyErr_Occurred())

Py_FatalError("can't initialize module stacktype");

}

Using C extension types in Python

Basic usage

% python

>>> import stacktype # load the type's module

>>> x = stacktype.Stack() # make a type-instance

>>> x.push('new') # call type-instance methods

>>> x # call the print handler

Sequence operators

>>> x[0] # stack_item

'new'

>>> x[1] # raise IndexError

Traceback (innermost last):

File "<stdin>", line 1, in ?

IndexError: out-of-bounds

>>> x[0:1] # stack_slice

>>> y = stacktype.Stack() # stacktype_new

>>> for c in 'SPAM': y.push(c) # stack_getattr ->

... # stack_push

>>> z = x + y # stack_concat

>>> z * 4 # stack_repeat

Comparisons, exceptions

>>> t = stacktype.Stack()

>>> t == y, t is y, t > y, t >= y # stack_compare

>>> for i in range(1000): y.push('hello' + `i`)

...

Traceback (innermost last):

File "<stdin>", line 1, in ?

stacktype.error: string-space overflow

Wrapping C extensions in Python

♦ In older Python releases, C types don’t support inheritance

♦ Wrapper classes add inheritance

♦ Wrappers can be specialized in Python

file: oopstack.py

import stacktype # get C type

class Stack:

def __init__(self, start=None): # wrap | make

self._base = start or stacktype.Stack()

def __getattr__(self, name):

return getattr(self._base, name) # attributes

def __cmp__(self, other):

return cmp(self._base, other)

def __repr__(self): # 'print'

print self._base,; return ''

def __add__(self, other): # operators

return Stack(self._base + other._base)

def __mul__(self, n):

return Stack(self._base * n) # new Stack

def __getitem__(self, i):

return self._base[i] # [i],in,for

def __len__(self):

return len(self._base)

file: substack.py

from oopstack import Stack # get wrapper class

class Substack(Stack):

def __init__(self, start=[]): # extend it in Python

Stack.__init__(self)

[. . .]

Writing extensions in C++

● Python is coded in portable ANSI C

● Normal C → C++ mixing rules apply

♦ Python header files

● Automatically wrapped in extern “C”, and may be included in C++ extensions freely

♦ Exported functions

● Wrap functions to be called by Python in extern “C” declarations

● Includes module initialization functions and (usually) method functions

♦ Static initializers

● C++ global or static object constructors (initializers) may not work correctly, if main program is linked by C

SWIG example (PP book)

SWIG is designed to export (‘wrap’) existing C/C++ components to Python programs. It generates complete extension modules with type conversion code based on C/C++ type signatures. It can also exprt C global variables, and do Python shadow class generation for C++ classes.

Also see “Extras” directory on the class CD for more examples

Module definition file

/* File : hellolib.i */

/******************************************************

* Swig module description file, for a C lib file.

* Generate by saying "swig -python hellolib.i".

* - %module sets name as known to Python importers

* - %{...%} encloses code added to wrapper verbatim

* - extern stmts declare exports in ANSI C syntax

* You could parse the whole header file by using a

* %include directive instead of the extern here,

* but externs let you select what is wrapped/exported;

* use '-Idir' swig args to specify .h search paths;

******************************************************/

%module hellowrap

#include <hellolib.h>

extern char *message(char*); /* or: %include "../HelloLib/hellolib.h" */

/* or: %include hellolib.h, and use -I arg */

Makefile, dynamic binding

############################################################

# Use SWIG to integrate the hellolib.c examples for use

# in Python programs. Using type signature information

# in .h files (or separate .i input files), SWIG generates

# the sort of logic we manually coded in the earlier example's

# hellolib_wrapper.c. SWIG creates hellolib_wrap.c when run;

# this makefile creates a hellowrap.so extension module file.

# To build, we run SWIG on hellolib.i, then compile and

# link with its output file. Note: you may need to first

# get and build the 'swig' executable if it's not already

# present on your machine: unpack, and run a './configure'

# and 'make', just like building Python from its source.

# You may also need to modify and source ./setup-swig.csh if

# you didn't 'make install' to put swig in standard places.

# See HelloLib/ makefiles for more details; the hellolib

# .c and .h files live in that dir, not here.

############################################################

# unless you've run make install

SWIG = ./myswig

PY = /home/mark/python1.5.2-ddjcd/Python-1.5.2

LIB = ../HelloLib

hellowrap.so: hellolib_wrap.o $(LIB)/hellolib.o

ld -shared hellolib_wrap.o $(LIB)/hellolib.o -o hellowrap.so

# generated wrapper module code

hellolib_wrap.o: hellolib_wrap.c $(LIB)/hellolib.h

gcc hellolib_wrap.c -c -g -I$(LIB) -I$(PY)/Include -I$(PY)

hellolib_wrap.c: hellolib.i

$(SWIG) -python -I$(LIB) hellolib.i

# C library code in another directory

$(LIB)/hellolib.o:

cd $(LIB); make -f makefile.hellolib-o hellolib.o

clean:

rm -f *.o *.so core

force:

rm -f *.o *.so core hellolib_wrap.c hellolib_wrap.doc

Python and rapid development

♦ Prototype-and-migrate

● Code in Python initially

● Move selected Python modules to C/C++

● Use profiler to pick components to migrate

♦ Seamless migration path

● C modules look just like Python modules

● C types look almost like Python classes

● Wrappers add inheritance to types

♦ Hybrid designs

● Python front-end, C/C++ back-end

● C/C++ application, Python customizations

The RAD slider

Prototyping with Python

Lab Session 11

Click here to go to lab exercises

Click here to go to exercise solutions

Click here to go to solution source files

Click here to go to lecture example files

Python in-process integration model

The ‘big picture’ revisited

Data conversions: Python →← C

SWIG example (PP book)

Lab Session 11