9. Built-in Tools Overview

 

 

 

 

¨  Python's tool box

·        Types and operations:   lists, dictionaries, files, slices,…

·        Functions:   len, range, zip, getattr…

·        Modules (Python and C):   string, os, Tkinter, pickle,…

·        Exceptions:   IndexError, KeyError,…

·        Attributes:   __dict__, __name__,…

·        Peripheral tools:   NumPy, SWIG, Jython, PythonWin,…

 

 

 

 

 

Topics

 

 

¨  Debugging options

¨  Testing frameworks

¨  Timing and profiling Python programs

¨  Packaging Python programs

¨  Installation tools

¨  Development tools for larger projects

¨  Summary: Python tool set layers

 

 

 

 

But first, the secret handshake…

 


 

 

 

Debugging options

 

 

¨    ‘pdb’ debugger: dbx-like command line interface

¨   Imported module, written in Python

¨   May also be run as script in recent releases

¨   Also see: ‘IDLE’ Tkinter-based debugger GUI  

¨   See library manuals for pdb commands and usage

 

 

 

 

Other error-handling tricks

 

·        Top-level stack tracebacks

·        Inserting print statements

·        Outer exception handlers

 

 

def safe(entry, *args):

    try:

        entry(*args)         # catch everything else (apply)

    except:

        import sys

        print sys.exc_info()[0], sys.exc_info()[1]    # type, value

 

  


 

Debugging example

 

 

 

file: boom.py

 

def func(x, y):

    return x / y

 

 

Session

 

>>> import boom

>>> boom.func(1, 0)

Traceback (innermost last):

  File "<stdin>", line 1, in ?

  File "boom.py", line 3, in func

    return x / y

ZeroDivisionError: integer division or modulo

>>> 

>>> import pdb

>>> pdb.run('boom.func(1, 0)')     Ü run/debug code

> <string>(0)?()

(Pdb) b boom.func                  Ü set breakpoint

(Pdb) c                            Ü continue program

> boom.py(2)func()

-> def func(x, y):

(Pdb) s                            Ü step 1 line

> boom.py(3)func()

-> return x / y

(Pdb) s

ZeroDivisionError: 'integer division or modulo'

> boom.py(3)func()

-> return x / y

(Pdb) where                        Ü stack trace

  <string>(1)?()

> boom.py(3)func()

-> return x / y

(Pdb) p y                          Ü print variables

0

 

 


 

Inspecting name-spaces

 

 

¨   Python lookups use 3-scope rule: local, global, built-in

¨   locals(), globals(): return name-spaces as dictionaries

 

% python

>>> def func(x):

...     a = 1

...     print locals()                # on function call

...     print globals().keys()

...

>>> class klass:

...     def __init__(self):

...         print locals()            # on instance creation

...         print globals().keys()

...     print locals()                # on class creation

...     print globals().keys()

...

{'__init__': <function __init__ at 76ed00>}

['__builtins__', '__name__', 'func', '__doc__']

 

>>> func(1)

{'a': 2, 'x': 1}

['__builtins__', '__name__', 'func', 'klass', '__doc__']

 

>>> x = klass()

{'self': <klass instance at 76f8d0>, 'arg': None}

['__builtins__', '__name__', 'func', 'klass', '__doc__']

 

>>> def nester(L, M, N):

...     class nested:               # assigns class to name

...         def __init__(self):

...             pass

...         print locals()          # local=class global=mod

...         print globals().keys()  # no access to L/M/N!

...     return nested               

...

>>> nester(1, 2, 3)

{'__init__': <function __init__ at 761e30>}

['__doc__', 'nester', '__name__', 'x', 'func', 'klass',...]

<class nested at 762960>


 

 

 

Dynamic coding tools

 

 

¨   apply    [now func(*args)] runs functions with argument tuples

¨   eval      evaluates a Python expression code-string

¨   exec      runs a Python statement code-string

¨   getattr fetches an object’s attribute by name string

¨   Supports run-time program construction

¨   Supports embedding Python in Python

 

 

 

Basic usage

 

>>> x = "2 ** 5"

>>> a = eval(x)

>>> a

32

 

>>> exec "print a / 2"

16

 

>>> def echo(a, b, c): print a, b, c

...

>>> apply(echo, (1, 2, 3))      # now: echo(*(1, 2, 3))

1 2 3

 

>>> import string

>>> getattr(string, "uppercase")

'ABCDEFGHIJKLMNOPQRSTUVWXYZ'

 

>>> D = {}

>>> exec "import math\nx = math.pi" in D, D

>>> D['x']

3.14159265359


 

Example

 

¨   Import module by name, run function by name/args

¨   Runs:  [ message.printer(‘sir’, ‘robin’) ]

¨   Also see:  new __import__ function

¨   Preview: will revisit model to embed Python in C

 

 

 

file: message.py

 

def printer(str1, str2):

    return 'brave', str1, str2

 

 

file: dynamic.py

 

def runFunction(moduleName, functionName, argsTuple):

    exec 'import ' + moduleName

    module = eval(moduleName)

    function = getattr(module, functionName)

    return function(*argsTuple)

 

if __name__ == '__main__':

    from sys import argv

    print runFunction(argv[1], argv[2], tuple(argv[3:]))

 

 

Command line

 

% python dynamic.py message printer sir robin

('brave', 'sir', 'robin')

 

 

 

 


 

Timing and profiling Python programs

 

 

¨   time module contains C library type tools

¨   Useful for performance tweaking (along with profiler)

¨   Warning: be careful to compare apples to apples!

Ø   See newer “timeit” module: easy, portable timing tools

 

 

 

 

Example: inline stack alternatives

 

 

¨   List based stacks

['spam0', 'spam1', 'spam2']  Ütop

 

·        Use list in-place changes: append/del

·        Lists resized on demand: grown in increments

 

 

 

¨   Tuple-pair based stacks

topÞ  ('spam2', ('spam1', ('spam0', None)))

 

·        Use tuple packing/unpacking assignments

·        Build a tree of 2-item tuples: (item, tree)

·        Like Lisp ‘cons’ cells/linked lists: avoids copies

 

 

 

 

 


 

 

file: testinline.py

#!/opt/local/bin/python

import time

from sys import argv, exit

numtests = 20

 

try:

    pushes, pops = eval(argv[1]), eval(argv[2])

except:

    print 'usage: testinline.py <pushes> <pops>'; exit(1)

 

def test(reps, func):

    start_cpu  = time.clock()

    for i in xrange(reps):                # call N times

        x = func()

    return time.clock() - start_cpu

 

def inline1():                            # builtin lists

    x = []

    for i in range(pushes): x.append('spam' + `i`)

    for i in range(pops):   del x[-1]

 

def inline2():                            # builtin tuples

    x = None

    for i in range(pushes): x = ('spam' + `i`, x)

    for i in range(pops):   (top, x) = x

 

print 'lists: ', test(numtests, inline1)  # run 20 times

print 'tuples:', test(numtests, inline2)

 

 

 

Results

 

% testinline.py 500 500     --20*(500 pushes + 500 pops)

lists:  0.77                --lists: append/del

tuples: 0.43                --tuples: pack/unpack

% testinline.py 1000 1000   --20K pushes + 20K pops

lists:  1.54

tuples: 0.87

% testinline.py 200 200

lists:  0.31

tuples: 0.17

% testinline.py 5000 5000

lists:  7.72

tuples: 4.5

 

 

 

Related Modules

 

 

datetime

 

>>> from datetime import datetime, timedelta

>>> x = datetime(2004, 11, 21)

>>> y = datetime(2005, 3, 19)

>>>

>>> y - x

datetime.timedelta(118)

>>>

>>> x + timedelta(30)

datetime.datetime(2004, 12, 21, 0, 0)

 

 

 

profile

 

·        Run as script or interactive, like pdb debugger

·        Warning: don’t run at IDLE prompt!

·        cProfile: more efficient version in 2.5+

·        pstats to report on results later

·        Optimizing: profile, time, shedskin/psyco, move to C

 

>>> import profile

>>> profile.run('import test1')

         35 function calls in 0.027 CPU seconds

 

   Ordered by: standard name

 

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)

       16    0.001    0.000    0.001    0.000 :0(range)

        1    0.005    0.005    0.005    0.005 :0(setprofile)

        1    0.002    0.002    0.022    0.022 <string>:1(?)

        1    0.000    0.000    0.027    0.027 profile:0(import test1)

        0    0.000             0.000          profile:0(profiler)

        1    0.000    0.000    0.020    0.020 test1.py:2(?)

       15    0.019    0.001    0.020    0.001 test1.py:2(myfun)

 

 

 

 

Example: timing iteration alternatives

 

 

 

 

File types and packaging options

 

 

Byte code

Modules compiled to portable byte-code on import: .pyc

compileall module forces imports, to make .pyc’s

.pyo files created and run with –O command-line flag

Frozen Binaries

Package byte-code + Python in an executable

Don’t require Python to be installed

Protect your program code

Other options

Import hooks support zip files, decryption, etc.

Pickler converts objects to/from text stream (serializer)

 

 

 

Format

Medium

Source files

.py files, scripts

Source files, no console

.pyw files (Windows,GUI)

Compiled byte-code files

.pyc files, .pyo files(1.5+)

C extensions on Windows

.pyd  (a .dll with init function)

Encrypted byte-code files

Import hooks, PyCrypto

Frozen binaries, self-installers

Py2Exe, PyInstaller, Freeze

distutils (see below)

Setup.py installation scripts

Zip files of modules

Auto in 2.4+ (zipimport 2.3)

Pickled objects

raw objects

Embedding mediums

databases, etc.

Jython: Java bytecode

network downloads, etc.

 

 

 

Development tools for larger projects

 

 

 

PyDoc

Displaying docstrings, program structure

PyChecker

Pre-run error checking (a “lint” for Python)

PyUnit

Unit testing framework (a.k.a. unittest)

Doctest

docstring-based regression test system

IDEs

IDLE, Komodo, PythonWin, PythonWorks

Profilers

profile, hotshot

Debuggers

pdb, IDLE point-and-click, print

Optimization

Psyco, .pyo bytecode, C extensions, SWIG

Packaging

Py2Exe, Installer, Freeze (above)

Distutils

packaging, install, build script system

Language tools

module packages, private attributes, class exceptions, __name__==__main__, docstrings

 

 

 

 

Testing tools in the standard library

 

 

 

Simplest convention

 

if __name__ == '__main__':

    unit test code here...

 

 


doctest module

·       automates interactive session

·       regression test system

 

# file spams.py

 

"""

This module works as follows:

 

>>> spams(3)

'spamspamspam'

>>> shrubbery()

'spamspamspamspam!!!'

"""

 

def spams(N):

    return 'spam' * N

 

def shrubbery():

    return spams(4) + '!!!'

 

if __name__ == '__main__':

    import doctest

    doctest.testmod()

 

 

 

C:\Python25>python spams.py -v

Trying:

    spams(3)

Expecting:

    'spamspamspam'

ok

Trying:

    shrubbery()

Expecting:

    'spamspamspamspam!!!'

ok

2 items had no tests:

    __main__.shrubbery

    __main__.spams

1 items passed all tests:

   2 tests in __main__

2 tests in 3 items.

2 passed and 0 failed.

Test passed.

 

 

 

unittest module (PyUnit)

·       class structure for unit test code

·       See library manual: test suites, …

 

import random

import unittest

 

class TestSequenceFunctions(unittest.TestCase):

   

    def setUp(self):

        self.seq = range(10)

 

    def testshuffle(self):

        # make sure the shuffled sequence does not lose any elements

        random.shuffle(self.seq)

        self.seq.sort()

        self.assertEqual(self.seq, range(10))

 

    def testchoice(self):

        element = random.choice(self.seq)

        self.assert_(element in self.seq)

 

    def testsample(self):

        self.assertRaises(ValueError, random.sample, self.seq, 20)

        for element in random.sample(self.seq, 5):

            self.assert_(element in self.seq)

 

if __name__ == '__main__':

    unittest.main()


 

Distutils: auto-install/build utility

 

 

 

 

# setup.py

 

from distutils.core import setup

setup(name='foo',

      version='1.0',

      py_modules=['foo'],

      )

 

 

 

 

# Usage

 

python setup.py sdist           # make source distribution

 

python setup.py install         # install the system

 

python setup.py bdist_wininst   # make Windows exe installer

 

python setup.py bdist_rpm       # make Linux RPM installer

 

python setup.py register        # register with PyPI site

 

 

 

 

# setup.py for C extension modules

 

from distutils.core import setup, Extension

setup(name='foo',

      version='1.0',

      ext_modules=[Extension('foo', ['foo.c'])],

      )


 

 

Summary: Python tool-set layers

 

 

 

 

¨  Built-ins

·        Lists, dictionaries, strings, library modules, etc.

·        High-level tools for simple, fast programming

      

 

 

¨  Python extensions

·        Functions, classes, modules

·        For adding extra features, and new object types

 

 

      

¨  C extensions

·        C modules, C types

·        For integrating external systems, optimizing components, customization

 

 

 

 

Lab Session 7

 

Click here to go to lab exercises

Click here to go to exercise solutions

Click here to go to solution source files

Click here to go to lecture example files