¨ Python's tool box
· Types and operations: lists, dictionaries, files, slices,…
· Functions: len, range, zip, getattr…
· Modules (Python and C): string, os, Tkinter, pickle,…
· Exceptions: IndexError, KeyError,…
· Attributes: __dict__, __name__,…
· Peripheral tools: NumPy, SWIG, Jython, PythonWin,…
Topics
¨ Debugging options
¨ Testing frameworks
¨ Timing and profiling Python programs
¨ Packaging Python programs
¨ Installation tools
¨ Development tools for larger projects
¨ Summary: Python tool set layers
But first, the secret handshake…
¨ ‘pdb’ debugger: dbx-like command line interface
¨ Imported module, written in Python
¨ May also be run as script in recent releases
¨ Also see: ‘IDLE’ Tkinter-based debugger GUI
¨ See library manuals for pdb commands and usage
Other error-handling tricks
· Top-level stack tracebacks
· Inserting print statements
· Outer exception handlers
def safe(entry, *args):
try:
entry(*args) # catch everything else (apply)
except:
import sys
print sys.exc_info()[0], sys.exc_info()[1] # type, value
Debugging example
file: boom.py
def func(x, y):
return x / y
Session
>>> import boom
>>> boom.func(1, 0)
Traceback (innermost last):
File "<stdin>", line 1, in ?
File "boom.py", line 3, in func
return x / y
ZeroDivisionError: integer division or modulo
>>>
>>> import pdb
>>> pdb.run('boom.func(1, 0)') Ü run/debug code
> <string>(0)?()
(Pdb) b boom.func Ü set breakpoint
(Pdb) c Ü continue program
> boom.py(2)func()
-> def func(x, y):
(Pdb) s Ü step 1 line
> boom.py(3)func()
-> return x / y
(Pdb) s
ZeroDivisionError: 'integer division or modulo'
> boom.py(3)func()
-> return x / y
(Pdb) where Ü stack trace
<string>(1)?()
> boom.py(3)func()
-> return x / y
(Pdb) p y Ü print variables
0
¨ Python lookups use 3-scope rule: local, global, built-in
¨ locals(), globals(): return name-spaces as dictionaries
% python
>>> def func(x):
... a = 1
... print locals() # on function call
... print globals().keys()
...
>>> class klass:
... def __init__(self):
... print locals() # on instance creation
... print globals().keys()
... print locals() # on class creation
... print globals().keys()
...
{'__init__': <function __init__ at 76ed00>}
['__builtins__', '__name__', 'func', '__doc__']
>>> func(1)
{'a': 2, 'x': 1}
['__builtins__', '__name__', 'func', 'klass', '__doc__']
>>> x = klass()
{'self': <klass instance at 76f8d0>, 'arg': None}
['__builtins__', '__name__', 'func', 'klass', '__doc__']
>>> def nester(L, M, N):
... class nested: # assigns class to name
... def __init__(self):
... pass
... print locals() # local=class global=mod
... print globals().keys() # no access to L/M/N!
... return nested
...
>>> nester(1, 2, 3)
{'__init__': <function __init__ at 761e30>}
['__doc__', 'nester', '__name__', 'x', 'func', 'klass',...]
<class nested at 762960>
¨ apply [now func(*args)] runs functions with argument tuples
¨ eval evaluates a Python expression code-string
¨ exec runs a Python statement code-string
¨ getattr fetches an object’s attribute by name string
¨ Supports run-time program construction
¨ Supports embedding Python in Python
Basic usage
>>> x = "2 ** 5"
>>> a = eval(x)
>>> a
32
>>> exec "print a / 2"
16
>>> def echo(a, b, c): print a, b, c
...
>>> apply(echo, (1, 2, 3)) # now: echo(*(1, 2, 3))
1 2 3
>>> import string
>>> getattr(string, "uppercase")
'ABCDEFGHIJKLMNOPQRSTUVWXYZ'
>>> D = {}
>>> exec "import math\nx = math.pi" in D, D
>>> D['x']
3.14159265359
Example
¨ Import module by name, run function by name/args
¨ Runs: [ message.printer(‘sir’, ‘robin’) ]
¨ Also see: new __import__ function
¨ Preview: will revisit model to embed Python in C
file: message.py
def printer(str1, str2):
return 'brave', str1, str2
file: dynamic.py
def runFunction(moduleName, functionName, argsTuple):
exec 'import ' + moduleName
module = eval(moduleName)
function = getattr(module, functionName)
return function(*argsTuple)
if __name__ == '__main__':
from sys import argv
print runFunction(argv[1], argv[2], tuple(argv[3:]))
Command line
% python dynamic.py message printer sir robin
('brave', 'sir', 'robin')
¨ time module contains C library type tools
¨ Useful for performance tweaking (along with profiler)
¨ Warning: be careful to compare apples to apples!
Ø See newer “timeit” module: easy, portable timing tools
Example: inline stack alternatives
¨ List based stacks
['spam0', 'spam1', 'spam2'] Ütop
·
Use list in-place changes: append/
· Lists resized on demand: grown in increments
¨ Tuple-pair based stacks
topÞ ('spam2',
('spam1', ('spam0', None)))
· Use tuple packing/unpacking assignments
· Build a tree of 2-item tuples: (item, tree)
· Like Lisp ‘cons’ cells/linked lists: avoids copies
file: testinline.py
#!/opt/local/bin/python
import time
from sys import argv, exit
numtests = 20
try:
pushes, pops = eval(argv[1]), eval(argv[2])
except:
print 'usage: testinline.py <pushes> <pops>'; exit(1)
def test(reps, func):
start_cpu = time.clock()
for i in xrange(reps): # call N times
x = func()
return time.clock() - start_cpu
def inline1(): # builtin lists
x = []
for i in range(pushes): x.append('spam' + `i`)
for i in range(pops): del x[-1]
def inline2(): # builtin tuples
x = None
for i in range(pushes): x = ('spam' + `i`, x)
for i in range(pops): (top, x) = x
print 'lists: ', test(numtests, inline1) # run 20 times
print 'tuples:', test(numtests, inline2)
Results
% testinline.py 500 500 --20*(500 pushes + 500 pops)
lists: 0.77 --lists: append/
tuples: 0.43 --tuples: pack/unpack
% testinline.py 1000 1000 --20K pushes + 20K pops
lists: 1.54
tuples: 0.87
% testinline.py 200 200
lists: 0.31
tuples: 0.17
% testinline.py 5000 5000
lists: 7.72
tuples: 4.5
Related Modules
datetime
>>> from datetime import datetime,
timedelta
>>> x = datetime(2004, 11, 21)
>>> y = datetime(2005, 3, 19)
>>>
>>> y - x
datetime.timedelta(118)
>>>
>>> x + timedelta(30)
datetime.datetime(2004, 12, 21, 0, 0)
profile
· Run as script or interactive, like pdb debugger
· Warning: don’t run at IDLE prompt!
· cProfile: more efficient version in 2.5+
· pstats to report on results later
· Optimizing: profile, time, shedskin/psyco, move to C
>>> import profile
>>> profile.run('import test1')
35 function calls in 0.027 CPU seconds
Ordered by: standard name
ncalls
tottime percall cumtime
percall filename:lineno(function)
16
0.001 0.000 0.001
0.000 :0(range)
1
0.005 0.005 0.005
0.005 :0(setprofile)
1
0.002 0.002 0.022
0.022 <string>:1(?)
1
0.000 0.000 0.027
0.027 profile:0(import test1)
0
0.000 0.000 profile:0(profiler)
1
0.000 0.000 0.020
0.020 test1.py:2(?)
15
0.019 0.001 0.020
0.001 test1.py:2(myfun)
Example: timing iteration alternatives
Byte code
Modules compiled to portable
byte-code on import: .pyc
compileall module forces
imports, to make .pyc’s
.pyo files created and run with
–O command-line flag
Frozen Binaries
Package byte-code + Python in
an executable
Don’t require Python to be
installed
Protect your program code
Other options
Import hooks support zip files,
decryption, etc.
Pickler converts objects
to/from text stream (serializer)
Format |
Medium |
Source files |
.py files, scripts |
Source files, no console |
.pyw files (Windows,GUI) |
Compiled byte-code files |
.pyc files, .pyo files(1.5+) |
C extensions on Windows |
.pyd (a .dll with
init function) |
Encrypted byte-code files |
Import hooks, PyCrypto |
Frozen binaries, self-installers |
Py2Exe, PyInstaller, Freeze |
distutils (see below) |
Setup.py installation scripts |
Zip files of modules |
Auto in 2.4+ (zipimport 2.3) |
Pickled objects |
raw objects |
Embedding mediums |
databases, etc. |
Jython: Java bytecode |
network downloads, etc. |
PyDoc |
Displaying docstrings, program structure |
PyChecker |
Pre-run error checking (a “lint” for Python) |
PyUnit |
Unit testing framework (a.k.a. unittest) |
Doctest |
docstring-based regression test system |
IDEs |
IDLE, Komodo, PythonWin, PythonWorks |
Profilers |
profile, hotshot |
Debuggers |
pdb, IDLE point-and-click, print |
Optimization |
Psyco, .pyo bytecode, C extensions, SWIG |
Packaging |
Py2Exe, Installer, Freeze (above) |
Distutils |
packaging, install, build script system |
Language tools |
module packages, private attributes, class exceptions,
__name__==__main__, docstrings |
Simplest convention
if __name__ == '__main__':
unit test code here...
doctest module
· automates interactive session
· regression test system
# file spams.py
"""
This module
works as follows:
>>>
spams(3)
'spamspamspam'
>>>
shrubbery()
'spamspamspamspam!!!'
"""
def spams(N):
return 'spam' * N
def
shrubbery():
return spams(4) + '!!!'
if __name__ ==
'__main__':
import doctest
doctest.testmod()
C:\Python25>python
spams.py -v
Trying:
spams(3)
Expecting:
'spamspamspam'
ok
Trying:
shrubbery()
Expecting:
'spamspamspamspam!!!'
ok
2 items had no
tests:
__main__.shrubbery
__main__.spams
1 items passed
all tests:
2 tests in __main__
2 tests in 3
items.
2 passed and 0
failed.
Test passed.
unittest module (PyUnit)
· class structure for unit test code
· See library manual: test suites, …
import random
import unittest
class
TestSequenceFunctions(unittest.TestCase):
def setUp(self):
self.seq = range(10)
def testshuffle(self):
# make sure the shuffled sequence does
not lose any elements
random.shuffle(self.seq)
self.seq.sort()
self.assertEqual(self.seq, range(10))
def testchoice(self):
element = random.choice(self.seq)
self.assert_(element in self.seq)
def testsample(self):
self.assertRaises(ValueError,
random.sample, self.seq, 20)
for element in random.sample(self.seq,
5):
self.assert_(element in self.seq)
if __name__ ==
'__main__':
unittest.main()
# setup.py
from distutils.core import setup
setup(name='foo',
version='1.0',
py_modules=['foo'],
)
# Usage
python setup.py sdist # make source distribution
python setup.py install # install the system
python setup.py bdist_wininst # make Windows exe installer
python setup.py bdist_rpm # make Linux RPM installer
python setup.py register # register with PyPI site
# setup.py for C extension modules
from distutils.core import setup, Extension
setup(name='foo',
version='1.0',
ext_modules=[Extension('foo', ['foo.c'])],
)
¨ Built-ins
· Lists, dictionaries, strings, library modules, etc.
· High-level tools for simple, fast programming
¨ Python extensions
· Functions, classes, modules
· For adding extra features, and new object types
¨ C extensions
· C modules, C types
· For integrating external systems, optimizing components, customization
Click here to go to
lab exercises
Click here to go to
exercise solutions
Click here to go to
solution source files
Click here to go to
lecture example files