16. Embedding Python in C/C++

 

 

 

 

¨  C/C++ runs Python code via API calls

¨  For dynamic system customization

 

 

 

 

 

Embedding topics

 

 

¨  Calling objects

¨  Running code strings

¨  Registration techniques

¨  Other topics: errors, tools, etc.

 


 

General embedding concepts

 

 

 

Embedded code forms

 

¨  Code strings

·        Running expressions, statements

 

¨  Callable objects

·        Calling functions, classes, methods

 

¨  Code files

·        Importing modules, executing scripts

 

 

 

 

Embedded code sources

 

 

Source

Description

Modules

fetching code by importing modules

Text files

fetching code from simple text files

Registration

letting Python pass code to a C extension

HTML tags

extracting code from web pages

Databases

fetching code from a database table

Processes

receiving code over sockets

Construction

building Python code in C at runtime

And so on

system registries, etc.


 

 

 

Common code sources

 

 

¨   Callable objects…    modules, registration

¨   Code strings…         files, registration, HTML,…

¨   Code files…             files, modules, scripts

 

 

 

 

Data communication techniques

 

 

 

Code form

Technique

Mode

Objects

function arguments

In/out

Objects

function return values

Out

Strings

expression results

Out

Strings, Objects

global module-level variables

In/out

Strings, Objects

C extension get/set functions

In/out

Strings, Objects

files, stdin/stdout, sockets,…

In/out

 

 

 

 


 

Running simple code strings

 

 

¨   ‘Py_Initialize’ initializes Python libraries

¨   ‘PyRun_SimpleString’ runs Python statements

¨   Runs in module ‘__main__’, no result from code

¨   Python code uses ‘environ’ extension module too

 

 

file: main1.c

#include <Python.h>

 

main(argc, argv)

int argc;

char **argv;

{

    /* This is the simplest embedding mode.  */

    /* Other API functions return results,   */

    /* accept namespace arguments, allow     */

    /* access to real Python objects, etc.   */

    /* Strings may be precompiled for speed. */

 

    Py_Initialize();                       /* init python */

    PyRun_SimpleString("print 'Hello embedded world!'");

 

    /* use C extension module above */

    PyRun_SimpleString("from environ import *");

    PyRun_SimpleString(

         "for i in range(4):\n"

             "\tprint i,\n"

             "\tprint 'Hello, %s' % getenv('USER')\n\n" );

 

    PyRun_SimpleString("print 'Bye embedded world!'");

}

 

 

 

 

 

Running the C program

 

 

 

% main1                                             

Hello embedded world!

0 Hello, mlutz

1 Hello, mlutz

2 Hello, mlutz

3 Hello, mlutz

Bye embedded world!

 

 

 

 

 

Same as typing at “>>>” prompt:

 

 

print 'Hello embedded world!'

from environ import *

for i in range(4):

    print i,

    print 'Hello, %s' % getenv('USER')

print 'Bye embedded world!

 

 

 


 

 

Building programs that embed Python

 

 

 

¨   Link with Python library (1), and your main()

¨   Add any libs referenced by Python lib (Modules/Setup)

¨   Older scheme (1.4): 4 Python .a libs + 2 Python .o files

 

 

 

 

file: Makefile.main1

# this file builds a C executable that embeds Python

# assuming no external module libs must be linked in

# works on Linux with a custom Python build tree

 

PY    = /home/mark/python1.5.2-ddjcd/Python-1.5.2

PYLIB = $(PY)/libpython1.5.a

PYINC = -I$(PY)/Include -I$(PY)

 

basic1: basic1.o

  cc basic1.o $(PYLIB) -g -export-dynamic -lm -ldl -o basic1

 

basic1.o: basic1.c

  cc basic1.c -c -g $(PYINC)

 

 


 

Calling objects and methods

 

 

 

The example task

 

¨   Make an instance of a Python class in a module file

¨   Call a method of that instance by name

¨   Python equivalent:

import <module>

object = <module>.<class>()

result = object.<method>(..args..)

 

 

 

 

Python API tools to be used

 

Tool

Description

PyObject*

The type of a generic Python object in C

PyImport_ImportModule

Imports a Python module, much like the Python ‘import’

PyObject_GetAttrString

Performs attribute qualifications: ‘object.name’, like ‘getattr’

PyEval_CallObject

Calls any callable object (class, function, method), like ‘apply’

Py_BuildValue

Converts C data to Python form, based on a format string

PyArg_Parse

Converts Python data to C form, based on a format string

Py_DECREF

Release ownership of object passed to C from the API


 

 

 

The task implementation

 

 

file: module.py   (on $PYTHONPATH)

 

class klass:

    def method(self, x, y):

        return "brave %s %s" % (x, y)   # run me from C

 

 

file: objects1.c

#include <Python.h>

#include <stdio.h>

 

main() {

  char *arg1="sir", *arg2="robin", *cstr;

  PyObject *pmod, *pclass, *pargs, *pinst, *pmeth, *pres;

 

  /* instance = module.klass() */

  Py_Initialize();

  pmod   = PyImport_ImportModule("module");

  pclass = PyObject_GetAttrString(pmod, "klass");

  Py_DECREF(pmod);

 

  pargs  = Py_BuildValue("()");

  pinst  = PyEval_CallObject(pclass, pargs);

  Py_DECREF(pclass);

  Py_DECREF(pargs);

 

  /* result = instance.method(x,y) */

  pmeth  = PyObject_GetAttrString(pinst, "method");

  Py_DECREF(pinst);

  pargs  = Py_BuildValue("(ss)", arg1, arg2);

  pres   = PyEval_CallObject(pmeth, pargs);

  Py_DECREF(pmeth);

  Py_DECREF(pargs);

 

  PyArg_Parse(pres, "s", &cstr);  /* convert to C */

  printf("%s\n", cstr);

  Py_DECREF(pres);

}

 

 

% objects1  

brave sir robin


 

With full error checking (!)

file: objects1err.c

#include <Python.h>

#include <stdio.h>

#define error(msg) do { printf("%s\n", msg); exit(1); } while (1)

 

main() {

  char *arg1="sir", *arg2="robin", *cstr;

  PyObject *pmod, *pclass, *pargs, *pinst, *pmeth, *pres;

 

  /* instance = module.klass() */

  Py_Initialize();

  pmod = PyImport_ImportModule("module");

  if (pmod == NULL)

      error("Can't load module");

  pclass = PyObject_GetAttrString(pmod, "klass");

  Py_DECREF(pmod);

  if (pclass == NULL)

      error("Can't get module.klass");

 

  pargs = Py_BuildValue("()");

  if (pargs == NULL) {

      Py_DECREF(pclass);

      error("Can't build arguments list");

  }

  pinst = PyEval_CallObject(pclass, pargs);

  Py_DECREF(pclass);

  Py_DECREF(pargs);

  if (pinst == NULL)

      error("Error calling module.klass()");

 

  /* result = instance.method(x,y) */

  pmeth  = PyObject_GetAttrString(pinst, "method");

  Py_DECREF(pinst);

  if (pmeth == NULL)

      error("Can't fetch klass.method");

  pargs = Py_BuildValue("(ss)", arg1, arg2);

  if (pargs == NULL) {

      Py_DECREF(pmeth);

      error("Can't build arguments list");

  }

  pres = PyEval_CallObject(pmeth, pargs); 

  Py_DECREF(pmeth);

  Py_DECREF(pargs);

  if (pres == NULL)

      error("Error calling klass.method");

 

  if (!PyArg_Parse(pres, "s", &cstr))                /* convert to C */

     error("Can't convert klass.method result");

  printf("%s\n", cstr);

  Py_DECREF(pres);

}


 

The easy way: an extended API

 

 

¨   Higher-level interface from “Programming Python

¨   Automates errors, reference counts, conversions

¨   Supports reloading and debugging embedded code

¨   Link with Python libs/objects + “pyembed*” files

 

 

 

file: objects2.c

#include <stdio.h>

#include "pyembed.h"

 

main () {

  int failflag;

  PyObject *pinst;

  char *arg1="sir", *arg2="robin", *cstr;

 

  failflag =

     Run_Function("module", "klass",    /* module.klass() */

                  "O", &pinst, "()")    /* result, args   */

     ||

     Run_Method(pinst, "method",        /* pinst.method() */

                  "s", &cstr,           /* result fmt/ptr */

                  "(ss)", arg1, arg2);  /* args fmt/values*/

 

  printf("%s\n", (!failflag) ? cstr : "Can't call objects");

  Py_XDECREF(pinst);

}

 



 

% objects2

brave sir robin

 


 

Running strings: results & name-spaces

 

 

¨     Runs: [ upper('spam') + '!' ] in string module

¨     Expression result comes back as a Python object

¨     ‘PyModule_GetDict’: module’s name-space dictionary

¨     ‘PyRun_String’: runs a code string in name-spaces

¨     Parse-mode flag: ‘Py_eval_input’ = expression

 

 

file codestring1.c

#include <Python.h>          /* standard API defs  */

 

main() {

    /* error checking omitted! */

    char *cstr;

    PyObject *pstr, *pmod, *pdict;

    Py_Initialize();

 

    /* result = string.upper('spam') + '!' */

    pmod  = PyImport_ImportModule("string"); /* namespace */

    pdict = PyModule_GetDict(pmod);

    pstr  = PyRun_String("upper('spam') + '!'",

                            Py_eval_input, pdict, pdict);

 

    /* convert result to C */

    PyArg_Parse(pstr, "s", &cstr);

    printf("%s\n", cstr);

    Py_DECREF(pmod);

    Py_DECREF(pstr);    /* free exported objects */

}

 

 

 

% codestring1

SPAM!


 

 

The easy way, part II: extended API

 

 

¨   Automates code strings, object calls, and methods

¨   Also supports running strings without modules

 

 

 

 

file: codestring2.c

#include "pyembed.h"

#include <stdio.h>

 

main() {

   char *cstr;

   int err =

      Run_Codestr(

         PY_EXPRESSION,                    /* expr|stmt?  */

         "upper('spam') + '!'", "string",  /* code,module */

         "s", &cstr);                      /* expr result */

   printf("%s\n", (!err) ? cstr : "Can't run string");

}

 

 

 

 

% codestring2

SPAM!

 


 

Other code string possibilities

 

 

¨   Making new dictionaries for string name-spaces

¨   Fetching strings from files, modules, HTML, databases

¨   Exporting C extension functions for communication

¨   Global variables for communication: ‘copy-in-copy-out’

 

 

Copy-in-copy-out

 

 

validate.py

 

# use QUANTITY, ORDERTYPE

# set QUANTITY, MESSAGES

[. . .]



C code…

 

    /* set input vars */

    char *string = . . .

    Set_Global("validate", "QUANTITY",  "i", quantity);

    Set_Global("validate", "ORDERTYPE", "s", ordertype);

 

    /* run code string */

    status = Run_Codestr(PY_STATEMENT,

                         string, "validate", "", NULL);

    if (status == -1)

        PyErr_Print();  /* stack traceback */

    else {

        /* fetch output vars */

        Get_Global("validate", "QUANTITY", "i", &quantity);

        Get_Global("validate", "MESSAGES", "s", &messages);

    }

  

Making namespace dictionaries

 

 

 

¨   Shamelessly stolen from “Programming Python

¨   Same as using built-in ‘eval’/‘exec’, ‘{}’, ‘X[key]’:

dict = {}

dict['Y'] = 2

exec 'X = 99' in dict, dict

exec 'X = X+Y' in dict, dict

print dict['X']                 # => 101

 

 

 

 

file: basic4.c

#include <Python.h>   

 

main() {

    int cval;

    PyObject *pdict, *pval;

    Py_Initialize();

  

    /* make a new namespace */

    pdict = PyDict_New();

    PyDict_SetItemString(pdict, "__builtins__",

                                PyEval_GetBuiltins());

    /* dict['Y'] = 2   */

    PyDict_SetItemString(pdict, "Y", PyInt_FromLong(2));  

   

    PyRun_String("X = 99",  Py_file_input, pdict, pdict);

    PyRun_String("X = X+Y", Py_file_input, pdict, pdict);

   

    /* fetch dict['X'] */

    pval = PyDict_GetItemString(pdict, "X");

    PyArg_Parse(pval, "i", &cval);   /* convert to C */

    printf("%d\n", cval);            /* result=101 */

    Py_DECREF(pdict);

}

 

 

 


 

Registering Python objects and strings

 

 

¨  A strategy for code location/source

¨  Python passes code to a C extension

¨  C saves the code and runs it later

¨  May register objects or code strings

¨  Works best if Python is ‘on top’

 

 

 

 

 

Components: extending + embedding

 

 

¨  A C extension module

·        Exports a registration function to Python (‘setHandler’)

 

¨  A C event routing function

·        Calls the registered Python object in response to events

 

¨  A Python client program

·        Registers functions, triggers events


Registration implementation

 

 

file: cregister.c

#include <Python.h>

#include <stdlib.h>

 

/***********************************************/

/* 1) code to route events to Python object    */

/* note that we could run strings here instead */

/***********************************************/

 

static PyObject *Handler = NULL;   /* Python object in C */

 

void Route_Event(char *label, int count)

{

    char *cres;

    PyObject *args, *pres;

 

    /* call Python handler */

    args = Py_BuildValue("(si)", label, count);

    pres = PyEval_CallObject(Handler, args);

    Py_DECREF(args);                       

 

    if (pres != NULL) {

        /* use and decref handler result */

        PyArg_Parse(pres, "s", &cres);

        printf("%s\n", cres);

        Py_DECREF(pres);

    }

}

 

/*****************************************************/

/* 2) python extension module to register handlers   */

/* python imports this module to set handler objects */

/*****************************************************/

 

static PyObject *

Register_Handler(PyObject *self, PyObject *args)

{

    /* save Python callable object */

    Py_XDECREF(Handler);                /* called before? */

    PyArg_Parse(args, "O", &Handler);   /* one argument?  */

    Py_XINCREF(Handler);                /* add reference  */

    Py_INCREF(Py_None);                 /* None=success   */

    return Py_None;

}

 

static PyObject *

Trigger_Event(PyObject *self, PyObject *args)

{

    /* let Python simulate event caught by C */

    static count = 0;

    Route_Event("spam", count++);

    Py_INCREF(Py_None); 

    return Py_None;

}

 

static struct PyMethodDef cregister_methods[] = {

    {"setHandler",    Register_Handler},    /* name, addr */

    {"triggerEvent",  Trigger_Event}, 

    {NULL, NULL}

};

 

void initcregister()       /* this is called by Python  */

{                          /* on first "import cregister" */

    (void) Py_InitModule("cregister", cregister_methods);

}

 

 

 


 

Python client program

 

 

¨   Combines extending and embedding

¨   Extending:     Python encloses C

¨   Embedding:   C calls Python on events

¨   C may also call embedded python code to register

 

 

file: register.py

 

# handle an event, return a result (or None)

 

def function1(label, count):

    return "%s number %i..." % (label, count)

 

def function2(label, count):

    return label * count

 

# register handlers, trigger events

 

import cregister

cregister.setHandler(function1)

for i in range(3):

    cregister.triggerEvent()   # simulate events caught by C

 

cregister.setHandler(function2)

for i in range(3):

    cregister.triggerEvent()   # routes events to function2

 

 

 

% python register.py

spam number 0...

spam number 1...

spam number 2...

spamspamspam

spamspamspamspam

spamspamspamspamspam


 

Registration tradeoffs

 

 

¨  +: Granularity

Associating actions with objects without external files

 

¨  -: Application structure

Requires Python on top, or extra embedding logic

 

¨  -: Complexity

Might require extra coding step to register code

 

¨  -: Reloading code

Difficult to reload without going through modules

 

 

Building the example

 

Solaris

# add the extension modules as dynamically linked modules;

# when first imported, they are located in a dir on

# $PYTHONPATH, and loaded into the process.

 

cregister.so: cregister.c

    $(Cc) cregister.c $(CFLAGS) -DDEBIG -KPIC -o cregister.o

    ld -G cregister.o -o $@

 

environ.so: environ.c

    $(Cc) environ.c $(CFLAGS) -KPIC -o environ.o

    ld -G environ.o -o $@

 

Linux

PY    = /home/mark/python1.5.2-ddjcd/Python-1.5.2

PYINC = -I$(PY)/Include -I$(PY)

cregister.so: cregister.c

     gcc cregister.c -g $(PYINC) -fpic -shared


 

 

 

Accessing C variables in Python

 

 

¨  Using an extension module 

import cvars           # import C variable wrapper module

cvars.setX(24)         # set C's X from Python

print cvars.getY()     # fetch C's Y from Python

 

 

¨  Using an extension type

import cvars           # import C variable wrapper module

c = cvars.interace()   # make interface object instance

c.X = (24)             # set attributes -> C variable

print c.Y              # get attributes -> C variable

 

 

¨  Using copy-in-copy-out

1. C copies X and Y values to variables in module M

2. C runs embedded code in module M’s name-space

3. C fetches the final values of the Python variables

 

 

 

 

C API equivalents in Python

 

¨   Python may be embedded in C or Python

¨   See section “Dynamic coding” for Python tools

 

C API tool

Python tool

PyImport_ImportModule

import

PyEval_CallObject

apply

PyRun_String

exec/eval

PyObject_GetAttrString

getattr


 

 

 

Running code files from C

 

 

 

¨  ‘PyRun_File’, ‘PyRun_SimpleFile’

¨  Module imports and reloads

¨  system, popen, fork/exec

 

 

 

 

 

Precompiling strings into byte-code

 

 

·        Modules are compiled once, on first import

·        Raw strings compiled when run, unless precompile

 

 

¨  Compile to byte code object

 

   PyCodeObject*

   Py_CompileString(char *string,

                    char *filename, int parsemode);

 

 

¨  Run byte-code object

 

   PyObject*

   PyEval_EvalCode(PyCodeObject *code,

                   PyObject *globalnamesdict,

                   PyObject *localnamesdict);

 


 

Embedding under C++

 

 

 

·        Python is coded in portable ANSI C

·        Normal C++ Þ C mixing rules apply

·        No special considerations when embedding

 

 

 

¨  Python header files

·        Automatically wrapped in extern “C”, and may be included in C++ programs freely

 

 

 

¨  Python libraries

·        Don’t need to be recompiled by the C++ compiler to link: API entry points are all extern “C”

 

 


 

More on object reference counts

 

 

 

¨   Python uses reference-count garbage collection

¨   New objects returned to C with a reference

¨   C must INCREF other objects to retain them

¨   C must DECREF objects it no longer needs

¨   XINCREF/XDECREF ignore NULL pointers

 

 

 

 

Resources:

 

 

¨   New API documentation

·        http://www.python.org/doc, Python/C API manual

 

 

¨   Older API documentation

·        http://www.python.org/doc/ext/ext.html

 

 

¨   Other references

·        Programming Python, Editions 2+

 

 

¨   Abstract object API

·        Tools in "abstract.h" header file are well-defined

 

 

¨   Extended API in chapter 15 of “Programming Python”

·        Run_Function, etc., handle most details

 

 

¨   Embedding examples in books, on the ‘net, etc.

·        http://rmi.net/~lutz/newex.html

 

 

¨   Check the C source code of the API

·        Not as hard as you may think!

 

 

 

 

 

 

 

Common API calls

 

 

New Name (1.3+)

INCREFs?

Python Equivalent

PyImport_AddModule

No

n/a

PyImport_ImportModule

Yes

import module

PyImport_ReloadModule

Yes

reload(module)

PyImport_GetModuleDict

No

sys.modules

PyModule_GetDict

No

module.__dict__

PyDict_GetItemString

No

dict[key]

PyDict_SetItemString

n/a

dict[key]=val

PyObject_GetAttrString

Yes

getattr(obj, attr)

PyObject_SetAttrString

n/a

setattr(obj, attr, val)

PyArg_ParseTuple

n/a

n/a

Py_BuildValue

Yes

n/a

PyEval_CallObject

Yes

apply(func, argtuple)

PyRun_String

Yes

eval(expr), exec stmt

PyErr_Print

n/a

traceback.print_exc

PyDict_New

Yes

{}


 

 

 

Integration error handling

 

 

 

 

¨  Extending: sending errors to Python

·        C extensions raise exceptions by returning NULL

·        ‘PyErr_SetString’ sets exception’s name and data

·        C may propagate error instead of setting

 

 

 

 

¨  Embedding: catching Python errors

·        API return values: NULL, or integer status codes

·        ‘PyErr_Fetch’ fetches exception info

·        ‘PyErr_Print’ shows Python stack traceback on stderr

·        Extended API functions return -1 or NULL on first error


 

 

Fetching exception information in C



 

file: pyerrors.c

#include <Python.h>

#include <stdio.h>

char save_error_type[1024], save_error_info[1024];

 

PyerrorHandler(char *msgFromC)

{

   /* process Python-related errors */

   /* call after Python API raises an exception */

 

   PyObject *errobj, *errdata, *errtraceback, *pystring;

   printf("%s\n", msgFromC);

 

   /* get latest python exception info */

   PyErr_Fetch(&errobj, &errdata, &errtraceback);

 

   pystring = NULL;

   if (errobj != NULL &&

      (pystring = PyObject_Str(errobj)) != NULL &&

      (PyString_Check(pystring))     

      )

       strcpy(save_error_type, PyString_AsString(pystring));

   else

       strcpy(save_error_type, "<unknown exception type>");

   Py_XDECREF(pystring);

 

   pystring = NULL;

   if (errdata != NULL &&

      (pystring = PyObject_Str(errdata)) != NULL &&

      (PyString_Check(pystring))

      )

       strcpy(save_error_info, PyString_AsString(pystring));

   else

       strcpy(save_error_info, "<unknown exception data>");

   Py_XDECREF(pystring);

 

   printf("%s\n%s\n", save_error_type, save_error_info);  

   Py_XDECREF(errobj);

   Py_XDECREF(errdata);         /* caller owns all 3 */

   Py_XDECREF(errtraceback);    /* already NULL'd out */

}

 

 

 

Automated integration tools

 

 

 

¨  SWIG

·        Generates Python interfaces to external C/C++ libraries

·        Uses C/C++ declarations and interface description files

·        For C++ class: generates C type + Python wrapper class

·        See unit 15, Programming Python 2nd edition, and http://www.swig.org/

 

¨  ILU

·        Implements CORBA distributed-object systems

·        Uses interface description files 

·        Platform and language independent

 

¨  Abstract object API

·        C API to create/process Python objects generically

·        In Python’s “abstract.h”: exports slicing, concatenation, etc.

 

¨  Embedded call API

·        C API to simplify common embedding tasks

·        In chapter 15 of “Programming Python” 1st Edition

 

¨  Modulator

·        Tkinter GUI: generates skeleton C module/type files

·        Users fill in the blanks with application-specific logic

 

¨  Other: ActiveX/COM interfaces


 

 

 

Modulator in action

 

 

 

 

 

 

 

 

 

 

 

Lab Session 12

 

Click here to go to lab exercises

Click here to go to exercise solutions

Click here to go to solution source files

 

Click here to go to lecture example files