16. Embedding Python in C/C++

 

 

 

 

     C/C++ runs Python code via API calls

     For dynamic system customization

 

 

 

 

 

Embedding topics

 

 

     Calling objects

     Running code strings

     Registration techniques

     Other topics: errors, tools, etc.

 


 

General embedding concepts

 

 

 

Embedded code forms

 

     Code strings

      Running expressions, statements

 

     Callable objects

      Calling functions, classes, methods

 

     Code files

      Importing modules, executing scripts

 

 

 

 

Embedded code sources

 

 

Source

Description

Modules

fetching code by importing modules

Text files

fetching code from simple text files

Registration

letting Python pass code to a C extension

HTML tags

extracting code from web pages

Databases

fetching code from a database table

Processes

receiving code over sockets

Construction

building Python code in C at runtime

And so on

system registries, etc.


 

 

 

Common code sources

 

 

     Callable objects…    modules, registration

     Code strings…         files, registration, HTML,…

     Code files…            files, modules, scripts

 

 

 

 

Data communication techniques

 

 

 

Code form

Technique

Mode

Objects

function arguments

In/out

Objects

function return values

Out

Strings

expression results

Out

Strings, Objects

global module-level variables

In/out

Strings, Objects

C extension get/set functions

In/out

Strings, Objects

files, stdin/stdout, sockets,…

In/out

 

 

 

 


 

Running simple code strings

 

 

     Py_Initialize’ initializes Python libraries

     PyRun_SimpleString’ runs Python statements

     Runs in module ‘__main__’, no result from code

     Python code uses ‘environ’ extension module too

 

 

file: main1.c

#include <Python.h>

 

main(argc, argv)

int argc;

char **argv;

{

    /* This is the simplest embedding mode.  */

    /* Other API functions return results,   */

    /* accept namespace arguments, allow     */

    /* access to real Python objects, etc.   */

    /* Strings may be precompiled for speed. */

 

    Py_Initialize();                       /* init python */

    PyRun_SimpleString("print 'Hello embedded world!'");

 

    /* use C extension module above */

    PyRun_SimpleString("from environ import *");

    PyRun_SimpleString(

         "for i in range(4):\n"

             "\tprint i,\n"

             "\tprint 'Hello, %s' % getenv('USER')\n\n" );

 

    PyRun_SimpleString("print 'Bye embedded world!'");

}

 

 

 

 

 

Running the C program

 

 

 

% main1                                             

Hello embedded world!

0 Hello, mlutz

1 Hello, mlutz

2 Hello, mlutz

3 Hello, mlutz

Bye embedded world!

 

 

 

 

 

Same as typing at “>>>” prompt:

 

 

print 'Hello embedded world!'

from environ import *

for i in range(4):

    print i,

    print 'Hello, %s' % getenv('USER')

print 'Bye embedded world!

 

 

 


 

 

Building programs that embed Python

 

 

 

     Link with Python library (1), and your main()

     Add any libs referenced by Python lib (Modules/Setup)

     Older scheme (1.4): 4 Python .a libs + 2 Python .o files

 

 

 

 

file: Makefile.main1

# this file builds a C executable that embeds Python

# assuming no external module libs must be linked in

# works on Linux with a custom Python build tree

 

PY    = /home/mark/python1.5.2-ddjcd/Python-1.5.2

PYLIB = $(PY)/libpython1.5.a

PYINC = -I$(PY)/Include -I$(PY)

 

basic1: basic1.o

  cc basic1.o $(PYLIB) -g -export-dynamic -lm -ldl -o basic1

 

basic1.o: basic1.c

  cc basic1.c -c -g $(PYINC)

 

 


 

Calling objects and methods

 

 

 

The example task

 

     Make an instance of a Python class in a module file

     Call a method of that instance by name

     Python equivalent:

import <module>

object = <module>.<class>()

result = object.<method>(..args..)

 

 

 

 

Python API tools to be used

 

Tool

Description

PyObject*

The type of a generic Python object in C

PyImport_ImportModule

Imports a Python module, much like the Python ‘import’

PyObject_GetAttrString

Performs attribute qualifications: ‘object.name’, like ‘getattr

PyEval_CallObject

Calls any callable object (class, function, method), like ‘apply’

Py_BuildValue

Converts C data to Python form, based on a format string

PyArg_Parse

Converts Python data to C form, based on a format string

Py_DECREF

Release ownership of object passed to C from the API


 

 

 

The task implementation

 

 

file: module.py   (on $PYTHONPATH)

 

class klass:

    def method(self, x, y):

        return "brave %s %s" % (x, y)   # run me from C

 

 

file: objects1.c

#include <Python.h>

#include <stdio.h>

 

main() {

  char *arg1="sir", *arg2="robin", *cstr;

  PyObject *pmod, *pclass, *pargs, *pinst, *pmeth, *pres;

 

  /* instance = module.klass() */

  Py_Initialize();

  pmod   = PyImport_ImportModule("module");

  pclass = PyObject_GetAttrString(pmod, "klass");

  Py_DECREF(pmod);

 

  pargs  = Py_BuildValue("()");

  pinst  = PyEval_CallObject(pclass, pargs);

  Py_DECREF(pclass);

  Py_DECREF(pargs);

 

  /* result = instance.method(x,y) */

  pmeth  = PyObject_GetAttrString(pinst, "method");

  Py_DECREF(pinst);

  pargs  = Py_BuildValue("(ss)", arg1, arg2);

  pres   = PyEval_CallObject(pmeth, pargs);

  Py_DECREF(pmeth);

  Py_DECREF(pargs);

 

  PyArg_Parse(pres, "s", &cstr);  /* convert to C */

  printf("%s\n", cstr);

  Py_DECREF(pres);

}

 

 

% objects1  

brave sir robin


 

 

With full error checking (!)

file: objects1err.c

#include <Python.h>

#include <stdio.h>

#define error(msg) do { printf("%s\n", msg); exit(1); } while (1)

 

main() {

  char *arg1="sir", *arg2="robin", *cstr;

  PyObject *pmod, *pclass, *pargs, *pinst, *pmeth, *pres;

 

  /* instance = module.klass() */

  Py_Initialize();

  pmod = PyImport_ImportModule("module");

  if (pmod == NULL)

      error("Can't load module");

  pclass = PyObject_GetAttrString(pmod, "klass");

  Py_DECREF(pmod);

  if (pclass == NULL)

      error("Can't get module.klass");

 

  pargs = Py_BuildValue("()");

  if (pargs == NULL) {

      Py_DECREF(pclass);

      error("Can't build arguments list");

  }

  pinst = PyEval_CallObject(pclass, pargs);

  Py_DECREF(pclass);

  Py_DECREF(pargs);

  if (pinst == NULL)

      error("Error calling module.klass()");

 

  /* result = instance.method(x,y) */

  pmeth  = PyObject_GetAttrString(pinst, "method");

  Py_DECREF(pinst);

  if (pmeth == NULL)

      error("Can't fetch klass.method");

  pargs = Py_BuildValue("(ss)", arg1, arg2);

  if (pargs == NULL) {

      Py_DECREF(pmeth);

      error("Can't build arguments list");

  }

  pres = PyEval_CallObject(pmeth, pargs); 

  Py_DECREF(pmeth);

  Py_DECREF(pargs);

  if (pres == NULL)

      error("Error calling klass.method");

 

  if (!PyArg_Parse(pres, "s", &cstr))                /* convert to C */

     error("Can't convert klass.method result");

  printf("%s\n", cstr);

  Py_DECREF(pres);

}


 

 

The easy way: an extended API

 

 

     Higher-level interface from “Programming Python

     Automates errors, reference counts, conversions

     Supports reloading and debugging embedded code

     Link with Python libs/objects + “pyembed*” files

 

 

 

file: objects2.c

#include <stdio.h>

#include "pyembed.h"

 

main () {

  int failflag;

  PyObject *pinst;

  char *arg1="sir", *arg2="robin", *cstr;

 

  failflag =

     Run_Function("module", "klass",    /* module.klass() */

                  "O", &pinst, "()")    /* result, args   */

     ||

     Run_Method(pinst, "method",        /* pinst.method() */

                  "s", &cstr,           /* result fmt/ptr */

                  "(ss)", arg1, arg2);  /* args fmt/values*/

 

  printf("%s\n", (!failflag) ? cstr : "Can't call objects");

  Py_XDECREF(pinst);

}

 



 

% objects2

brave sir robin

 


 

Running strings: results & name-spaces

 

 

      Runs: [ upper('spam') + '!' ] in string module

      Expression result comes back as a Python object

      PyModule_GetDict’: module’s name-space dictionary

      PyRun_String’: runs a code string in name-spaces

      Parse-mode flag: ‘Py_eval_input’ = expression

 

 

file codestring1.c

#include <Python.h>          /* standard API defs  */

 

main() {

    /* error checking omitted! */

    char *cstr;

    PyObject *pstr, *pmod, *pdict;

    Py_Initialize();

 

    /* result = string.upper('spam') + '!' */

    pmod  = PyImport_ImportModule("string"); /* namespace */

    pdict = PyModule_GetDict(pmod);

    pstr  = PyRun_String("upper('spam') + '!'",

                            Py_eval_input, pdict, pdict);

 

    /* convert result to C */

    PyArg_Parse(pstr, "s", &cstr);

    printf("%s\n", cstr);

    Py_DECREF(pmod);

    Py_DECREF(pstr);    /* free exported objects */

}

 

 

 

% codestring1

SPAM!


 

 

The easy way, part II: extended API

 

 

     Automates code strings, object calls, and methods

     Also supports running strings without modules

 

 

 

 

file: codestring2.c

#include "pyembed.h"

#include <stdio.h>

 

main() {

   char *cstr;

   int err =

      Run_Codestr(

         PY_EXPRESSION,                    /* expr|stmt?  */

         "upper('spam') + '!'", "string",  /* code,module */

         "s", &cstr);                      /* expr result */

   printf("%s\n", (!err) ? cstr : "Can't run string");

}

 

 

 

 

% codestring2

SPAM!

 


 

Other code string possibilities

 

 

     Making new dictionaries for string name-spaces

     Fetching strings from files, modules, HTML, databases

     Exporting C extension functions for communication

     Global variables for communication: ‘copy-in-copy-out’

 

 

Copy-in-copy-out

 

 

validate.py

 

# use QUANTITY, ORDERTYPE

# set QUANTITY, MESSAGES

[. . .]



C code…

 

    /* set input vars */

    char *string = . . .

    Set_Global("validate", "QUANTITY",  "i", quantity);

    Set_Global("validate", "ORDERTYPE", "s", ordertype);

 

    /* run code string */

    status = Run_Codestr(PY_STATEMENT,

                         string, "validate", "", NULL);

    if (status == -1)

        PyErr_Print();  /* stack traceback */

    else {

        /* fetch output vars */

        Get_Global("validate", "QUANTITY", "i", &quantity);

        Get_Global("validate", "MESSAGES", "s", &messages);

    }

  

 

 

Making namespace dictionaries

 

 

 

     Shamelessly stolen from “Programming Python

     Same as using built-in ‘eval’/‘exec’, ‘{}’, ‘X[key]’:

dict = {}

dict['Y'] = 2

exec 'X = 99' in dict, dict

exec 'X = X+Y' in dict, dict

print dict['X']                 # => 101

 

 

 

 

file: basic4.c

#include <Python.h>   

 

main() {

    int cval;

    PyObject *pdict, *pval;

    Py_Initialize();

  

    /* make a new namespace */

    pdict = PyDict_New();

    PyDict_SetItemString(pdict, "__builtins__",

                                PyEval_GetBuiltins());

    /* dict['Y'] = 2   */

    PyDict_SetItemString(pdict, "Y", PyInt_FromLong(2));  

   

    PyRun_String("X = 99",  Py_file_input, pdict, pdict);

    PyRun_String("X = X+Y", Py_file_input, pdict, pdict);

   

    /* fetch dict['X'] */

    pval = PyDict_GetItemString(pdict, "X");

    PyArg_Parse(pval, "i", &cval);   /* convert to C */

    printf("%d\n", cval);            /* result=101 */

    Py_DECREF(pdict);

}

 

 

 


 

Registering Python objects and strings

 

 

     A strategy for code location/source

     Python passes code to a C extension

     C saves the code and runs it later

     May register objects or code strings

     Works best if Python is ‘on top’

 

 

 

 

 

Components: extending + embedding

 

 

     A C extension module

      Exports a registration function to Python (‘setHandler’)

 

     A C event routing function

      Calls the registered Python object in response to events

 

     A Python client program

      Registers functions, triggers events


Registration implementation

 

 

file: cregister.c

#include <Python.h>

#include <stdlib.h>

 

/***********************************************/

/* 1) code to route events to Python object    */

/* note that we could run strings here instead */

/***********************************************/

 

static PyObject *Handler = NULL;   /* Python object in C */

 

void Route_Event(char *label, int count)

{

    char *cres;

    PyObject *args, *pres;

 

    /* call Python handler */

    args = Py_BuildValue("(si)", label, count);

    pres = PyEval_CallObject(Handler, args);

    Py_DECREF(args);                       

 

    if (pres != NULL) {

        /* use and decref handler result */

        PyArg_Parse(pres, "s", &cres);

        printf("%s\n", cres);

        Py_DECREF(pres);

    }

}

 

/*****************************************************/

/* 2) python extension module to register handlers   */

/* python imports this module to set handler objects */

/*****************************************************/

 

static PyObject *

Register_Handler(PyObject *self, PyObject *args)

{

    /* save Python callable object */

    Py_XDECREF(Handler);                /* called before? */

    PyArg_Parse(args, "O", &Handler);   /* one argument?  */

    Py_XINCREF(Handler);                /* add reference  */

    Py_INCREF(Py_None);                 /* None=success   */

    return Py_None;

}

 

static PyObject *

Trigger_Event(PyObject *self, PyObject *args)

{

    /* let Python simulate event caught by C */

    static count = 0;

    Route_Event("spam", count++);

    Py_INCREF(Py_None); 

    return Py_None;

}

 

static struct PyMethodDef cregister_methods[] = {

    {"setHandler",    Register_Handler},    /* name, addr */

    {"triggerEvent",  Trigger_Event}, 

    {NULL, NULL}

};

 

void initcregister()       /* this is called by Python  */

{                          /* on first "import cregister" */

    (void) Py_InitModule("cregister", cregister_methods);

}

 

 

 


 

Python client program

 

 

     Combines extending and embedding

     Extending:     Python encloses C

     Embedding:   C calls Python on events

     C may also call embedded python code to register

 

 

file: register.py

 

# handle an event, return a result (or None)

 

def function1(label, count):

    return "%s number %i..." % (label, count)

 

def function2(label, count):

    return label * count

 

# register handlers, trigger events

 

import cregister

cregister.setHandler(function1)

for i in range(3):

    cregister.triggerEvent()   # simulate events caught by C

 

cregister.setHandler(function2)

for i in range(3):

    cregister.triggerEvent()   # routes events to function2

 

 

 

% python register.py

spam number 0...

spam number 1...

spam number 2...

spamspamspam

spamspamspamspam

spamspamspamspamspam


 

Registration tradeoffs

 

 

     +: Granularity

Associating actions with objects without external files

 

     -: Application structure

Requires Python on top, or extra embedding logic

 

     -: Complexity

Might require extra coding step to register code

 

     -: Reloading code

Difficult to reload without going through modules

 

 

Building the example

 

Solaris

# add the extension modules as dynamically linked modules;

# when first imported, they are located in a dir on

# $PYTHONPATH, and loaded into the process.

 

cregister.so: cregister.c

    $(Cc) cregister.c $(CFLAGS) -DDEBIG -KPIC -o cregister.o

    ld -G cregister.o -o $@

 

environ.so: environ.c

    $(Cc) environ.c $(CFLAGS) -KPIC -o environ.o

    ld -G environ.o -o $@

 

Linux

PY    = /home/mark/python1.5.2-ddjcd/Python-1.5.2

PYINC = -I$(PY)/Include -I$(PY)

cregister.so: cregister.c

     gcc cregister.c -g $(PYINC) -fpic -shared


 

 

 

Accessing C variables in Python

 

 

     Using an extension module 

import cvars           # import C variable wrapper module

cvars.setX(24)         # set C's X from Python

print cvars.getY()     # fetch C's Y from Python

 

 

     Using an extension type

import cvars           # import C variable wrapper module

c = cvars.interace()   # make interface object instance

c.X = (24)             # set attributes -> C variable

print c.Y              # get attributes -> C variable

 

 

     Using copy-in-copy-out

1. C copies X and Y values to variables in module M

2. C runs embedded code in module M’s name-space

3. C fetches the final values of the Python variables

 

 

 

 

C API equivalents in Python

 

     Python may be embedded in C or Python

     See section “Dynamic coding” for Python tools

 

C API tool

Python tool

PyImport_ImportModule

import

PyEval_CallObject

apply

PyRun_String

exec/eval

PyObject_GetAttrString

getattr


 

 

 

Running code files from C

 

 

 

     PyRun_File’, ‘PyRun_SimpleFile

     Module imports and reloads

     system, popen, fork/exec

 

 

 

 

 

Precompiling strings into byte-code

 

 

      Modules are compiled once, on first import

      Raw strings compiled when run, unless precompile

 

 

     Compile to byte code object

 

   PyCodeObject*

   Py_CompileString(char *string,

                    char *filename, int parsemode);

 

 

     Run byte-code object

 

   PyObject*

   PyEval_EvalCode(PyCodeObject *code,

                   PyObject *globalnamesdict,

                   PyObject *localnamesdict);

 


 

Embedding under C++

 

 

 

      Python is coded in portable ANSI C

      Normal C++ C mixing rules apply

      No special considerations when embedding

 

 

 

     Python header files

      Automatically wrapped in extern “C”, and may be included in C++ programs freely

 

 

 

     Python libraries

      Don’t need to be recompiled by the C++ compiler to link: API entry points are all extern “C”

 

 


 

More on object reference counts

 

 

 

     Python uses reference-count garbage collection

     New objects returned to C with a reference

     C must INCREF other objects to retain them

     C must DECREF objects it no longer needs

     XINCREF/XDECREF ignore NULL pointers

 

 

 

 

Resources:

 

 

     New API documentation

      http://www.python.org/doc, Python/C API manual

 

 

     Older API documentation

      http://www.python.org/doc/ext/ext.html

 

 

     Other references

      Programming Python, Editions 2+

 

 

     Abstract object API

      Tools in "abstract.h" header file are well-defined

 

 

     Extended API in chapter 15 of “Programming Python”

      Run_Function, etc., handle most details

 

 

     Embedding examples in books, on the ‘net, etc.

      http://rmi.net/~lutz/newex.html

 

 

     Check the C source code of the API

      Not as hard as you may think!

 

 

 

 

 

 

 

Common API calls

 

 

New Name (1.3+)

INCREFs?

Python Equivalent

PyImport_AddModule

No

n/a

PyImport_ImportModule

Yes

import module

PyImport_ReloadModule

Yes

reload(module)

PyImport_GetModuleDict

No

sys.modules

PyModule_GetDict

No

module.__dict__

PyDict_GetItemString

No

dict[key]

PyDict_SetItemString

n/a

dict[key]=val

PyObject_GetAttrString

Yes

getattr(obj, attr)

PyObject_SetAttrString

n/a

setattr(obj, attr, val)

PyArg_ParseTuple

n/a

n/a

Py_BuildValue

Yes

n/a

PyEval_CallObject

Yes

apply(func, argtuple)

PyRun_String

Yes

eval(expr), exec stmt

PyErr_Print

n/a

traceback.print_exc

PyDict_New

Yes

{}


 

 

 

Integration error handling

 

 

 

 

     Extending: sending errors to Python

      C extensions raise exceptions by returning NULL

      PyErr_SetString’ sets exception’s name and data

      C may propagate error instead of setting

 

 

 

 

     Embedding: catching Python errors

      API return values: NULL, or integer status codes

      PyErr_Fetch’ fetches exception info

      PyErr_Print’ shows Python stack traceback on stderr

      Extended API functions return -1 or NULL on first error


 

 

Fetching exception information in C



 

file: pyerrors.c

#include <Python.h>

#include <stdio.h>

char save_error_type[1024], save_error_info[1024];

 

PyerrorHandler(char *msgFromC)

{

   /* process Python-related errors */

   /* call after Python API raises an exception */

 

   PyObject *errobj, *errdata, *errtraceback, *pystring;

   printf("%s\n", msgFromC);

 

   /* get latest python exception info */

   PyErr_Fetch(&errobj, &errdata, &errtraceback);

 

   pystring = NULL;

   if (errobj != NULL &&

      (pystring = PyObject_Str(errobj)) != NULL &&

      (PyString_Check(pystring))     

      )

       strcpy(save_error_type, PyString_AsString(pystring));

   else

       strcpy(save_error_type, "<unknown exception type>");

   Py_XDECREF(pystring);

 

   pystring = NULL;

   if (errdata != NULL &&

      (pystring = PyObject_Str(errdata)) != NULL &&

      (PyString_Check(pystring))

      )

       strcpy(save_error_info, PyString_AsString(pystring));

   else

       strcpy(save_error_info, "<unknown exception data>");

   Py_XDECREF(pystring);

 

   printf("%s\n%s\n", save_error_type, save_error_info);  

   Py_XDECREF(errobj);

   Py_XDECREF(errdata);         /* caller owns all 3 */

   Py_XDECREF(errtraceback);    /* already NULL'd out */

}

 

 

 

Automated integration tools

 

 

 

     SWIG

      Generates Python interfaces to external C/C++ libraries

      Uses C/C++ declarations and interface description files

      For C++ class: generates C type + Python wrapper class

      See unit 15, the book Programming Python editions 2+, and http://www.swig.org/

 

     ILU

      Implements CORBA distributed-object systems

      Uses interface description files 

      Platform and language independent

 

     Abstract object API

      C API to create/process Python objects generically

      In Python’s “abstract.h”: exports slicing, concatenation, etc.

 

     Embedded call API

      C API to simplify common embedding tasks

      In “Programming Python”, editions 1..3

 

     Modulator

      Tkinter GUI: generates skeleton C module/type files

      Users fill in the blanks with application-specific logic

      An old idea, which has been revived recently (forgetting history much?)

 

     Other: ActiveX/COM interfaces


 

 

 

Modulator in action (old)

 

 

 

Description: Description: modulat2

 

 

 

 

 

 

 

 

Lab Session 12

 

Click here to go to lab exercises

Click here to go to exercise solutions

Click here to go to solution source files

 

Click here to go to lecture example files