­­­9. Built-in To­­­ols Overview





     Python's tool box

      Types and operations:   lists, dictionaries, files, slices,…

      Functions:   len, range, zip, getattr

      Modules (Python and C):   string, os, Tkinter, pickle,…

      Exceptions:   IndexError, KeyError,…

      Attributes:   __dict__, __name__,…

      Peripheral tools:   NumPy, SWIG, Jython, PythonWin,…









     Debugging options

     Testing frameworks

     Timing and profiling Python programs

     Packaging Python programs

     Installation tools

     Development tools for larger projects

     Summary: Python tool set layers





But first, the secret handshake…





Debugging options



     pdb’ debugger: dbx-like command line interface

     Imported module, written in Python

     May also be run as script →  python -m pdb script.py

     Also see: ‘IDLE’ Tkinter-based debugger GUI 

     See library manuals for pdb commands and usage





Other error-handling tricks


      Error messages

      Top-level stack tracebacks

      Inserting print statements

      Outer exception handlers



def safe(entry, *args):


        entry(*args)         # catch everything else (apply)


        import sys

        print sys.exc_info()[0], sys.exc_info()[1]    # type, value




Debugging example




file: boom.py


def func(x, y):

    return x / y





>>> import boom

>>> boom.func(1, 0)

Traceback (innermost last):

  File "<stdin>", line 1, in ?

  File "boom.py", line 3, in func

    return x / y

ZeroDivisionError: integer division or modulo


>>> import pdb

>>> pdb.run('boom.func(1, 0)')     # run/debug code

> <string>(0)?()

(Pdb) b boom.func                  # set breakpoint

(Pdb) c                            # continue program

> boom.py(2)func()

-> def func(x, y):

(Pdb) s                            # step 1 line

> boom.py(3)func()

-> return x / y

(Pdb) s

ZeroDivisionError: 'integer division or modulo'

> boom.py(3)func()

-> return x / y

(Pdb) where                        # stack trace


> boom.py(3)func()

-> return x / y

(Pdb) p y                          # print variables





Inspecting name-spaces



     Python lookups use 3-scope rule: local, global, built-in

     locals(), globals(): return name-spaces as dictionaries


% python

>>> def func(x):

...     a = 1

...     print locals()                # on function call

...     print globals().keys()


>>> class klass:

...     def __init__(self):

...         print locals()            # on instance creation

...         print globals().keys()

...     print locals()                # on class creation

...     print globals().keys()


{'__init__': <function __init__ at 76ed00>}

['__builtins__', '__name__', 'func', '__doc__']


>>> func(1)

{'a': 2, 'x': 1}

['__builtins__', '__name__', 'func', 'klass', '__doc__']


>>> x = klass()

{'self': <klass instance at 76f8d0>, 'arg': None}

['__builtins__', '__name__', 'func', 'klass', '__doc__']


>>> def nester(L, M, N):

...     class nested:               # assigns class to name

...         def __init__(self):

...             pass

...         print locals()          # local=class global=mod

...         print globals().keys()  # no access to L/M/N!

...     return nested               


>>> nester(1, 2, 3)

{'__init__': <function __init__ at 761e30>}

['__doc__', 'nester', '__name__', 'x', 'func', 'klass',...]

<class nested at 762960>




Dynamic coding tools



     apply    [now func(*args)] runs functions with argument tuples

     eval      evaluates a Python expression code-string

     exec      runs a Python statement code-string (3.X: exec())

     getattr fetches an object’s attribute by name string

     Supports run-time program construction

     Supports embedding Python in Python




Basic usage


>>> x = "2 ** 5"

>>> a = eval(x)

>>> a



>>> exec "print a / 2"         # 3.X: exec('print(a / 2)')



>>> def echo(a, b, c): print a, b, c


>>> apply(echo, (1, 2, 3))     # now: echo(*(1, 2, 3))

1 2 3


>>> import string

>>> getattr(string, "uppercase")



>>> D = {}

>>> exec "import math\nx = math.pi" in D, D   # 3.X exec()

>>> D['x']







     Import module by name, run function by name/args

     Runs:  [ message.printer(‘sir’, ‘robin’) ]

     Also see:  newer __import__ function

     Preview: will revisit model to embed Python in C




file: message.py


def printer(str1, str2):

    return 'brave', str1, str2



file: dynamic.py


def runFunction(moduleName, functionName, argsTuple):

    exec 'import ' + moduleName

    module = eval(moduleName)

    function = getattr(module, functionName)

    return function(*argsTuple)


if __name__ == '__main__':

    from sys import argv

    print runFunction(argv[1], argv[2], tuple(argv[3:]))



Command line


% python dynamic.py message printer sir robin

('brave', 'sir', 'robin')






Timing and profiling Python programs



     time module contains C library type tools

     Useful for performance tweaking (along with profiler)

     Warning: be careful to compare apples to apples!


See also newer “timeit” module: automated, portable timing tools

>>> import timeit

>>> min(timeit.repeat(stmt="[x ** 2 for x in range(1000)]", number=1000, repeat=5))



c:\code> python -m timeit -n 1000 -r 5 "[x ** 2 for x in range(1000)]"

1000 loops, best of 5: 505 usec per loop





Example: inline stack alternatives



     List based stacks

['spam0', 'spam1', 'spam2']  top


      Use list in-place changes: append/del

      Lists resized on demand: grown in increments




     Tuple-pair based stacks

top  ('spam2', ('spam1', ('spam0', None)))


      Use tuple packing/unpacking assignments

      Build a tree of 2-item tuples: (item, tree)

      Like Lisp ‘cons’ cells/linked lists: avoids copies





file: testinline.py


import time

from sys import argv, exit

numtests = 20



    pushes, pops = eval(argv[1]), eval(argv[2])


    print 'usage: testinline.py <pushes> <pops>'; exit(1)


def test(reps, func):

    start_cpu  = time.clock()

    for i in xrange(reps):                # call N times

        x = func()

    return time.clock() - start_cpu


def inline1():                            # builtin lists

    x = []

    for i in range(pushes): x.append('spam' + `i`)

    for i in range(pops):   del x[-1]


def inline2():                            # builtin tuples

    x = None

    for i in range(pushes): x = ('spam' + `i`, x)

    for i in range(pops):   (top, x) = x


print 'lists: ', test(numtests, inline1)  # run 20 times

print 'tuples:', test(numtests, inline2)




Results (on an ancient machine…)


% testinline.py 500 500     --20*(500 pushes + 500 pops)

lists:  0.77                --lists: append/del

tuples: 0.43                --tuples: pack/unpack

% testinline.py 1000 1000   --20K pushes + 20K pops

lists:  1.54

tuples: 0.87

% testinline.py 200 200

lists:  0.31

tuples: 0.17

% testinline.py 5000 5000

lists:  7.72

tuples: 4.5




Related Modules





>>> from datetime import datetime, timedelta

>>> x = datetime(2004, 11, 21)

>>> y = datetime(2005, 3, 19)


>>> y - x



>>> x + timedelta(30)

datetime.datetime(2004, 12, 21, 0, 0)






      Run as script or interactive, like pdb debugger

      As script python -m profile script.py

      Warning: don’t run at IDLE prompt (too much data)!

      cProfile: more efficient version in 2.5+

      pstats to report on results later

      Optimizing: profile, time, shedskin/psyco, move to C


>>> import profile

>>> profile.run('import test1')

         35 function calls in 0.027 CPU seconds


   Ordered by: standard name


   ncalls  tottime  percall  cumtime  percall filename:lineno(function)

       16    0.001    0.000    0.001    0.000 :0(range)

        1    0.005    0.005    0.005    0.005 :0(setprofile)

        1    0.002    0.002    0.022    0.022 <string>:1(?)

        1    0.000    0.000    0.027    0.027 profile:0(import test1)

        0    0.000             0.000          profile:0(profiler)

        1    0.000    0.000    0.020    0.020 test1.py:2(?)

       15    0.019    0.001    0.020    0.001 test1.py:2(myfun)





Example: timing iteration alternatives





File types and packaging options



Byte code

      Modules compiled to portable byte-code on import: .pyc

      compileall module forces imports, to make .pyc’s

      .pyo files created and run with –O command-line flag (removed in 3.5!)

Frozen Binaries

      Package byte-code + Python in an executable

      Don’t require Python to be installed

      Protect your program code

Other options

      Import hooks support zip files, decryption, etc.

      Pickler converts objects to/from text stream (serializer)






Source files

.py files, scripts

Source files, no console

.pyw files (Windows,GUI)

Compiled byte-code files

.pyc files, .pyo files(1.5 ~  3.4)

C extensions on Windows

.pyd  (a .dll with init function)

Encrypted byte-code files

Import hooks, PyCrypto

Frozen binaries, self-installers

Py2Exe, PyInstaller, cx_freeze

distutils (see below), pip (newer)

Setup.py installation scripts

Zip files of modules

Auto in 2.4+ (zipimport 2.3+)

Pickled objects

raw objects

Embedding mediums

databases, etc.

Jython: Java bytecode

network downloads, etc.




Development tools for larger projects





Displaying docstrings, program structure


Pre-run error checking (a “lint” for Python)


Unit testing framework (a.k.a. unittest)


docstring-based regression test system


IDLE, Komodo, PythonWin, PythonWorks


profile, hotshot


pdb, IDLE point-and-click, print


Psyco, .pyo bytecode, C extensions, SWIG


Py2Exe, Installer, Freeze (above)


packaging, install, build script system

Language tools

module packages, private attributes, class exceptions, __name__==__main__, docstrings





Testing tools in the standard library



See also third-party testing tools, such as Nose



Simplest convention


if __name__ == '__main__':

    unit test code here...



doctest module

    automates interactive session

    regression test system


# file spams.py



This module works as follows:


>>> spams(3)


>>> shrubbery()




def spams(N):

    return 'spam' * N


def shrubbery():

    return spams(4) + '!!!'


if __name__ == '__main__':

    import doctest





C:\Python25>python spams.py -v











2 items had no tests:



1 items passed all tests:

   2 tests in __main__

2 tests in 3 items.

2 passed and 0 failed.

Test passed.




unittest module (PyUnit)

    class structure for unit test code

    See library manual: test suites, …


import random

import unittest


class TestSequenceFunctions(unittest.TestCase):


    def setUp(self):

        self.seq = range(10)


    def testshuffle(self):

        # make sure the shuffled sequence does not lose any elements



        self.assertEqual(self.seq, range(10))


    def testchoice(self):

        element = random.choice(self.seq)

        self.assert_(element in self.seq)


    def testsample(self):

        self.assertRaises(ValueError, random.sample, self.seq, 20)

        for element in random.sample(self.seq, 5):

            self.assert_(element in self.seq)


if __name__ == '__main__':



Distutils: auto-install/build utility



See also more recent “pip” installer regime, and the PyPI site




# setup.py


from distutils.core import setup









# Usage


python setup.py sdist           # make source distribution


python setup.py install         # install the system


python setup.py bdist_wininst   # make Windows exe installer


python setup.py bdist_rpm       # make Linux RPM installer


python setup.py register        # register with PyPI site





# setup.py for C extension modules


from distutils.core import setup, Extension



      ext_modules=[Extension('foo', ['foo.c'])],




Summary: Python tool-set layers






      Lists, dictionaries, strings, library modules, etc.

      High-level tools for simple, fast programming




     Python extensions

      Functions, classes, modules

      For adding extra features, and new object types




     C extensions

      C modules, C types

      For integrating external systems, optimizing components, customization





Lab Session 7


Click here to go to lab exercises

Click here to go to exercise solutions

Click here to go to solution source files

Click here to go to lecture example files