♦ C/C++ runs Python code via API calls
♦ For dynamic system customization
Embedding topics
♦ Calling objects
♦ Running code strings
♦ Registration techniques
♦ Other topics: errors, tools, etc.
General
embedding concepts
Embedded code forms
♦ Code strings
● Running expressions, statements
♦ Callable objects
● Calling functions, classes, methods
♦ Code files
● Importing modules, executing scripts
Embedded code sources
|
Source |
Description |
|
Modules |
fetching code by importing modules |
|
Text
files |
fetching code from simple text files |
|
Registration
|
letting Python pass code to a C
extension |
|
HTML
tags |
extracting code from web pages |
|
Databases |
fetching code from a database table |
|
Processes
|
receiving code over sockets |
|
Construction
|
building Python code in C at runtime |
|
And
so on |
system registries, etc. |
Common code sources
♦ Callable objects… modules, registration
♦ Code strings… files, registration, HTML,…
♦ Code files… files, modules, scripts
Data communication techniques
|
Code form |
Technique |
Mode |
|
Objects |
function arguments |
In/out |
|
Objects |
function return values |
Out |
|
Strings |
expression results |
Out |
|
Strings, Objects |
global module-level variables |
In/out |
|
Strings, Objects |
C extension get/set functions |
In/out |
|
Strings, Objects |
files, stdin/stdout, sockets,… |
In/out |
Running
simple code strings
♦ ‘Py_Initialize’ initializes Python libraries
♦ ‘PyRun_SimpleString’ runs Python statements
♦ Runs in module ‘__main__’, no result from code
♦ Python code uses ‘environ’ extension module too
file: main1.c
#include <Python.h>
main(argc, argv)
int argc;
char **argv;
{
/* This is the simplest embedding mode. */
/* Other API functions return results, */
/* accept namespace arguments, allow */
/* access to real Python objects, etc. */
/* Strings may be precompiled for speed. */
Py_Initialize(); /* init python */
PyRun_SimpleString("print 'Hello embedded world!'");
/* use C extension module above */
PyRun_SimpleString("from environ import *");
PyRun_SimpleString(
"for i in range(4):\n"
"\tprint i,\n"
"\tprint 'Hello, %s' % getenv('USER')\n\n" );
PyRun_SimpleString("print 'Bye embedded world!'");
}
Running the C program
% main1
Hello embedded world!
0 Hello, mlutz
1 Hello, mlutz
2 Hello, mlutz
3 Hello, mlutz
Bye embedded world!
Same as typing at “>>>” prompt:
print 'Hello embedded world!'
from environ import *
for i
in range(4):
print i,
print 'Hello, %s'
% getenv('USER')
print 'Bye embedded world!
Building programs that embed Python
♦ Link with Python library (1), and your main()
♦ Add any libs referenced by Python lib (Modules/Setup)
♦ Older scheme (1.4): 4 Python .a libs + 2 Python .o files
file: Makefile.main1
# this file builds a C executable that embeds Python
# assuming no external module libs must be linked in
# works on Linux with a custom Python build tree
PY = /home/mark/python1.5.2-ddjcd/Python-1.5.2
PYLIB = $(PY)/libpython1.5.a
PYINC = -I$(PY)/Include -I$(PY)
basic1: basic1.o
cc basic1.o $(PYLIB) -g -export-dynamic -lm -ldl -o basic1
basic1.o: basic1.c
cc basic1.c -c -g $(PYINC)
Calling
objects and methods
The
example task
♦ Make an instance of a Python class in a module file
♦ Call a method of that instance by name
♦ Python equivalent:
import <module>
object = <module>.<class>()
result = object.<method>(..args..)
Python API tools to be used
|
Tool |
Description |
|
PyObject* |
The type of a generic Python object in C |
|
PyImport_ImportModule |
Imports a Python module, much like the Python ‘import’ |
|
PyObject_GetAttrString |
Performs attribute qualifications: ‘object.name’, like ‘getattr’ |
|
PyEval_CallObject |
Calls any callable object (class, function, method), like
‘apply’ |
|
Py_BuildValue |
Converts C data to Python form, based on a format string |
|
PyArg_Parse |
Converts Python data to C form, based on a format string |
|
Py_DECREF |
Release ownership of object passed to C from the API |
The task implementation
file: module.py (on $PYTHONPATH)
class klass:
def method(self, x, y):
return "brave %s %s" % (x, y) # run me from C
file: objects1.c
#include <Python.h>
#include <stdio.h>
main() {
char *arg1="sir", *arg2="robin", *cstr;
PyObject *pmod, *pclass, *pargs, *pinst, *pmeth, *pres;
/* instance = module.klass() */
Py_Initialize();
pmod = PyImport_ImportModule("module");
pclass = PyObject_GetAttrString(pmod, "klass");
Py_DECREF(pmod);
pargs = Py_BuildValue("()");
pinst = PyEval_CallObject(pclass, pargs);
Py_DECREF(pclass);
Py_DECREF(pargs);
/* result = instance.method(x,y) */
pmeth = PyObject_GetAttrString(pinst, "method");
Py_DECREF(pinst);
pargs = Py_BuildValue("(ss)", arg1, arg2);
pres = PyEval_CallObject(pmeth, pargs);
Py_DECREF(pmeth);
Py_DECREF(pargs);
PyArg_Parse(pres, "s", &cstr); /* convert to C */
printf("%s\n", cstr);
Py_DECREF(pres);
}
% objects1
brave sir robin
With full error
checking (!)
file: objects1err.c
#include
<Python.h>
#include
<stdio.h>
#define
error(msg) do { printf("%s\n", msg);
exit(1); } while (1)
main() {
char
*arg1="sir", *arg2="robin", *cstr;
PyObject *pmod, *pclass, *pargs, *pinst, *pmeth, *pres;
/* instance = module.klass() */
Py_Initialize();
pmod
= PyImport_ImportModule("module");
if (pmod == NULL)
error("Can't
load module");
pclass
= PyObject_GetAttrString(pmod,
"klass");
Py_DECREF(pmod);
if (pclass == NULL)
error("Can't
get module.klass");
pargs
= Py_BuildValue("()");
if (pargs == NULL) {
Py_DECREF(pclass);
error("Can't
build arguments list");
}
pinst
= PyEval_CallObject(pclass,
pargs);
Py_DECREF(pclass);
Py_DECREF(pargs);
if (pinst == NULL)
error("Error
calling module.klass()");
/* result = instance.method(x,y) */
pmeth = PyObject_GetAttrString(pinst,
"method");
Py_DECREF(pinst);
if (pmeth == NULL)
error("Can't
fetch klass.method");
pargs
= Py_BuildValue("(ss)",
arg1, arg2);
if (pargs == NULL) {
Py_DECREF(pmeth);
error("Can't
build arguments list");
}
pres
= PyEval_CallObject(pmeth, pargs);
Py_DECREF(pmeth);
Py_DECREF(pargs);
if (pres == NULL)
error("Error
calling klass.method");
if (!PyArg_Parse(pres, "s",
&cstr)) /* convert to C */
error("Can't
convert klass.method result");
printf("%s\n", cstr);
Py_DECREF(pres);
}
The easy way: an extended API
♦ Higher-level interface from “Programming Python”
♦ Automates errors, reference counts, conversions
♦ Supports reloading and debugging embedded code
♦ Link with Python libs/objects + “pyembed*” files
file: objects2.c
#include <stdio.h>
#include "pyembed.h"
main () {
int failflag;
PyObject *pinst;
char *arg1="sir", *arg2="robin", *cstr;
failflag =
Run_Function("module", "klass", /* module.klass() */
"O", &pinst, "()") /* result, args */
||
Run_Method(pinst, "method", /* pinst.method() */
"s", &cstr, /* result fmt/ptr */
"(ss)", arg1, arg2); /* args fmt/values*/
printf("%s\n", (!failflag) ? cstr : "Can't call objects");
Py_XDECREF(pinst);
}
% objects2
brave sir robin
Running
strings: results & name-spaces
♦ Runs: [ upper('spam') + '!' ] in string
module
♦ Expression result comes back as a
Python object
♦ ‘PyModule_GetDict’:
module’s name-space dictionary
♦ ‘PyRun_String’:
runs a code string in name-spaces
♦ Parse-mode flag: ‘Py_eval_input’
= expression
file codestring1.c
#include <Python.h> /* standard API defs */
main() {
/* error checking omitted! */
char *cstr;
PyObject *pstr, *pmod, *pdict;
Py_Initialize();
/* result = string.upper('spam') + '!' */
pmod = PyImport_ImportModule("string"); /* namespace */
pdict = PyModule_GetDict(pmod);
pstr = PyRun_String("upper('spam') + '!'",
Py_eval_input, pdict, pdict);
/* convert result to C */
PyArg_Parse(pstr, "s", &cstr);
printf("%s\n", cstr);
Py_DECREF(pmod);
Py_DECREF(pstr); /* free exported objects */
}
% codestring1
SPAM!
The easy way, part II: extended
API
♦ Automates code strings, object calls, and methods
♦ Also supports running strings without modules
file: codestring2.c
#include "pyembed.h"
#include <stdio.h>
main() {
char *cstr;
int err =
Run_Codestr(
PY_EXPRESSION, /* expr|stmt? */
"upper('spam') + '!'", "string", /* code,module */
"s", &cstr); /* expr result */
printf("%s\n", (!err) ? cstr : "Can't run string");
}
% codestring2
SPAM!
Other
code string possibilities
♦ Making new dictionaries for string name-spaces
♦ Fetching strings from files, modules, HTML, databases
♦ Exporting C extension functions for communication
♦ Global variables for communication: ‘copy-in-copy-out’
Copy-in-copy-out
validate.py
# use QUANTITY, ORDERTYPE
# set QUANTITY, MESSAGES
[. . .]
C code…
/* set input vars */
char *string = . . .
Set_Global("validate", "QUANTITY", "i", quantity);
Set_Global("validate", "ORDERTYPE", "s", ordertype);
/* run code string */
status = Run_Codestr(PY_STATEMENT,
string, "validate", "", NULL);
if (status == -1)
PyErr_Print(); /* stack traceback */
else {
/* fetch output vars */
Get_Global("validate", "QUANTITY", "i", &quantity);
Get_Global("validate", "MESSAGES", "s", &messages);
}
Making namespace dictionaries
♦ Shamelessly stolen from “Programming Python”
♦ Same as using built-in ‘eval’/‘exec’, ‘{}’, ‘X[key]’:
dict = {}
dict['Y'] = 2
exec 'X = 99' in dict, dict
exec 'X = X+Y' in dict, dict
print dict['X'] # => 101
file: basic4.c
#include <Python.h>
main() {
int cval;
PyObject *pdict, *pval;
Py_Initialize();
/* make a new namespace */
pdict = PyDict_New();
PyDict_SetItemString(pdict, "__builtins__",
PyEval_GetBuiltins());
/* dict['Y'] = 2 */
PyDict_SetItemString(pdict, "Y", PyInt_FromLong(2));
PyRun_String("X = 99", Py_file_input, pdict, pdict);
PyRun_String("X = X+Y", Py_file_input, pdict, pdict);
/* fetch dict['X'] */
pval = PyDict_GetItemString(pdict, "X");
PyArg_Parse(pval, "i", &cval); /* convert to C */
printf("%d\n", cval); /* result=101 */
Py_DECREF(pdict);
}
Registering
Python objects and strings
♦ A strategy for code location/source
♦ Python passes code to a C extension
♦ C saves the code and runs it later
♦ May register objects or code strings
♦ Works best if Python is ‘on top’
Components: extending + embedding
♦ A C extension module
● Exports a registration function to Python (‘setHandler’)
♦ A C event routing function
● Calls the registered Python object in response to events
♦ A Python client program
● Registers functions, triggers events
Registration implementation
file: cregister.c
#include <Python.h>
#include <stdlib.h>
/***********************************************/
/* 1) code to route events to Python object */
/* note that we could run strings here instead */
/***********************************************/
static PyObject *Handler = NULL; /* Python object in C */
void Route_Event(char *label, int count)
{
char *cres;
PyObject *args, *pres;
/* call Python handler */
args = Py_BuildValue("(si)", label, count);
pres = PyEval_CallObject(Handler, args);
Py_DECREF(args);
if (pres != NULL) {
/* use and decref handler result */
PyArg_Parse(pres, "s", &cres);
printf("%s\n", cres);
Py_DECREF(pres);
}
}
/*****************************************************/
/* 2) python extension module to register handlers */
/* python imports this module to set handler objects */
/*****************************************************/
static PyObject *
Register_Handler(PyObject *self, PyObject *args)
{
/* save Python callable object */
Py_XDECREF(Handler); /* called before? */
PyArg_Parse(args, "O", &Handler); /* one argument? */
Py_XINCREF(Handler); /* add reference */
Py_INCREF(Py_None); /* None=success */
return Py_None;
}
static PyObject *
Trigger_Event(PyObject *self, PyObject *args)
{
/* let Python simulate event caught by C */
static count = 0;
Route_Event("spam", count++);
Py_INCREF(Py_None);
return Py_None;
}
static struct PyMethodDef cregister_methods[] = {
{"setHandler", Register_Handler}, /* name, addr */
{"triggerEvent", Trigger_Event},
{NULL, NULL}
};
void initcregister() /* this is called by Python */
{ /* on first "import cregister" */
(void) Py_InitModule("cregister", cregister_methods);
}
Python client program
♦ Combines extending and embedding
♦ Extending: Python encloses C
♦ Embedding: C calls Python on events
♦ C may also call embedded python code to register
file: register.py
# handle an event, return a result (or None)
def function1(label, count):
return "%s number %i..." % (label, count)
def function2(label, count):
return label * count
# register handlers, trigger events
import cregister
cregister.setHandler(function1)
for i in range(3):
cregister.triggerEvent() # simulate events caught by C
cregister.setHandler(function2)
for i in range(3):
cregister.triggerEvent() # routes events to function2
% python register.py
spam number 0...
spam number 1...
spam number 2...
spamspamspam
spamspamspamspam
spamspamspamspamspam
Registration tradeoffs
♦ +: Granularity
Associating actions with objects without external files
♦ -: Application structure
Requires Python on top, or extra embedding logic
♦ -: Complexity
Might require extra coding step to register code
♦ -: Reloading code
Difficult to reload without going through modules
Building the example
Solaris
# add the extension modules as dynamically linked modules;
# when first imported, they are located in a dir on
# $PYTHONPATH, and loaded into the process.
cregister.so: cregister.c
$(Cc) cregister.c $(CFLAGS) -DDEBIG -KPIC -o cregister.o
ld -G cregister.o -o $@
environ.so: environ.c
$(Cc) environ.c $(CFLAGS) -KPIC -o environ.o
ld -G environ.o -o $@
Linux
PY = /home/mark/python1.5.2-ddjcd/Python-1.5.2
PYINC = -I$(PY)/Include -I$(PY)
cregister.so: cregister.c
gcc cregister.c -g $(PYINC) -fpic -shared
Accessing
C variables in Python
♦ Using an extension module
import cvars # import C variable wrapper module
cvars.setX(24) # set C's X from Python
print cvars.getY() # fetch C's Y from Python
♦ Using an extension type
import cvars # import C variable wrapper module
c = cvars.interace() # make interface object instance
c.X = (24) # set attributes -> C variable
print c.Y # get attributes -> C variable
♦ Using copy-in-copy-out
1. C copies X and Y values to variables in module M
2. C runs embedded code in module M’s name-space
3. C fetches the final values of the Python variables
C API
equivalents in Python
♦ Python may be embedded in C or Python
♦ See section “Dynamic coding” for Python tools
|
C API tool |
Python tool |
|
PyImport_ImportModule |
import |
|
PyEval_CallObject |
apply |
|
PyRun_String |
exec/eval |
|
PyObject_GetAttrString |
getattr |
Running
code files from C
♦ ‘PyRun_File’, ‘PyRun_SimpleFile’
♦ Module imports and reloads
♦ system, popen, fork/exec
Precompiling
strings into byte-code
● Modules are compiled once, on first import
● Raw strings compiled when run, unless precompile
♦ Compile to byte code object
PyCodeObject*
Py_CompileString(char *string,
char *filename, int parsemode);
♦ Run byte-code object
PyObject*
PyEval_EvalCode(PyCodeObject *code,
PyObject *globalnamesdict,
PyObject *localnamesdict);
Embedding
under C++
● Python is coded in portable ANSI C
● Normal C++ → C mixing rules apply
● No special considerations when embedding
♦ Python header files
● Automatically wrapped in extern “C”, and may be included in C++ programs freely
♦ Python libraries
● Don’t need to be recompiled by the C++ compiler to link: API entry points are all extern “C”
More on
object reference counts
♦ Python uses reference-count garbage collection
♦ New objects returned to C with a reference
♦ C must INCREF other objects to retain them
♦ C must DECREF objects it no longer needs
♦ XINCREF/XDECREF ignore NULL pointers
Resources:
♦ New API documentation
● http://www.python.org/doc, Python/C API manual
♦ Older API documentation
● http://www.python.org/doc/ext/ext.html
♦ Other references
● Programming Python, Editions 2+
♦ Abstract object API
● Tools in "abstract.h" header file are well-defined
♦ Extended API in chapter 15 of “Programming Python”
● Run_Function, etc., handle most details
♦ Embedding examples in books, on the ‘net, etc.
● http://rmi.net/~lutz/newex.html
♦ Check the C source code of the API
● Not as hard as you may think!
Common API calls
|
New Name (1.3+) |
INCREFs? |
Python Equivalent |
|
PyImport_AddModule |
No |
n/a |
|
PyImport_ImportModule |
Yes |
import module |
|
PyImport_ReloadModule |
Yes |
reload(module) |
|
PyImport_GetModuleDict |
No |
sys.modules |
|
PyModule_GetDict |
No |
module.__dict__ |
|
PyDict_GetItemString |
No |
dict[key] |
|
PyDict_SetItemString |
n/a |
dict[key]=val |
|
PyObject_GetAttrString |
Yes |
getattr(obj, attr) |
|
PyObject_SetAttrString |
n/a |
setattr(obj, attr, val) |
|
PyArg_ParseTuple |
n/a |
n/a |
|
Py_BuildValue |
Yes |
n/a |
|
PyEval_CallObject |
Yes |
apply(func, argtuple) |
|
PyRun_String |
Yes |
eval(expr),
exec stmt |
|
PyErr_Print |
n/a |
traceback.print_exc |
|
PyDict_New |
Yes |
{} |
Integration
error handling
♦ Extending: sending errors to Python
● C extensions raise exceptions by returning NULL
● ‘PyErr_SetString’ sets exception’s name and data
● C may propagate error instead of setting
♦ Embedding: catching Python errors
● API return values: NULL, or integer status codes
● ‘PyErr_Fetch’ fetches exception info
● ‘PyErr_Print’ shows Python stack traceback on stderr
● Extended API functions return -1 or NULL on first error
Fetching exception information in
C
file: pyerrors.c
#include <Python.h>
#include <stdio.h>
char save_error_type[1024], save_error_info[1024];
PyerrorHandler(char *msgFromC)
{
/* process Python-related errors */
/* call after Python API raises an exception */
PyObject *errobj, *errdata, *errtraceback, *pystring;
printf("%s\n", msgFromC);
/* get latest python exception info */
PyErr_Fetch(&errobj, &errdata, &errtraceback);
pystring = NULL;
if (errobj != NULL &&
(pystring = PyObject_Str(errobj)) != NULL &&
(PyString_Check(pystring))
)
strcpy(save_error_type, PyString_AsString(pystring));
else
strcpy(save_error_type, "<unknown exception type>");
Py_XDECREF(pystring);
pystring = NULL;
if (errdata != NULL &&
(pystring = PyObject_Str(errdata)) != NULL &&
(PyString_Check(pystring))
)
strcpy(save_error_info, PyString_AsString(pystring));
else
strcpy(save_error_info, "<unknown exception data>");
Py_XDECREF(pystring);
printf("%s\n%s\n", save_error_type, save_error_info);
Py_XDECREF(errobj);
Py_XDECREF(errdata); /* caller owns all 3 */
Py_XDECREF(errtraceback); /* already NULL'd out */
}
Automated
integration tools
♦ SWIG
● Generates Python interfaces to external C/C++ libraries
● Uses C/C++ declarations and interface description files
● For C++ class: generates C type + Python wrapper class
● See unit 15, the book Programming Python editions 2+, and http://www.swig.org/
♦ ILU
● Implements CORBA distributed-object systems
● Uses interface description files
● Platform and language independent
♦ Abstract object API
● C API to create/process Python objects generically
● In Python’s “abstract.h”: exports slicing, concatenation, etc.
♦ Embedded call API
● C API to simplify common embedding tasks
● In “Programming Python”, editions 1..3
♦ Modulator
● Tkinter GUI: generates skeleton C module/type files
● Users fill in the blanks with application-specific logic
● An old idea, which has been revived recently (forgetting history much?)
♦ Other: ActiveX/COM interfaces
Modulator in action (old)
Lab Session 12
Click here to go to
lab exercises
Click here to go to
exercise solutions
Click here to go to solution source files
Click here to go to
lecture example files