♦ Python's tool box
● Types and operations: lists, dictionaries, files, slices,…
● Functions: len, range, zip, getattr…
● Modules (Python and C): string, os, Tkinter, pickle,…
● Exceptions: IndexError, KeyError,…
● Attributes: __dict__, __name__,…
● Peripheral tools: NumPy, SWIG, Jython, PythonWin,…
Topics
♦ Debugging options
♦ Testing frameworks
♦ Timing and profiling Python programs
♦ Packaging Python programs
♦ Installation tools
♦ Development tools for larger projects
♦ Summary: Python tool set layers
But first, the secret handshake…
♦ ‘pdb’ debugger: dbx-like command line interface
♦ Imported module, written in Python
♦ May also be run as script → python -m pdb script.py
♦ Also see: ‘IDLE’ Tkinter-based debugger GUI
♦ See library manuals for pdb commands and usage
Other error-handling tricks
● Error messages
● Top-level stack tracebacks
● Inserting print statements
● Outer exception handlers
def safe(entry, *args):
try:
entry(*args) # catch everything else (apply)
except:
import sys
print sys.exc_info()[0], sys.exc_info()[1] # type, value
Debugging example
file: boom.py
def func(x, y):
return x / y
Session
>>> import boom
>>> boom.func(1, 0)
Traceback (innermost last):
File "<stdin>", line 1, in ?
File "boom.py", line 3, in func
return x / y
ZeroDivisionError: integer division or modulo
>>>
>>> import pdb
>>> pdb.run('boom.func(1, 0)') # run/debug code
> <string>(0)?()
(Pdb) b boom.func # set breakpoint
(Pdb) c # continue program
> boom.py(2)func()
-> def func(x, y):
(Pdb) s # step 1 line
> boom.py(3)func()
-> return x / y
(Pdb) s
ZeroDivisionError: 'integer division or modulo'
> boom.py(3)func()
-> return x / y
(Pdb) where # stack trace
<string>(1)?()
> boom.py(3)func()
-> return x / y
(Pdb) p y # print variables
0
♦ Python lookups use 3-scope rule: local, global, built-in
♦ locals(), globals(): return name-spaces as dictionaries
% python
>>> def func(x):
... a = 1
... print locals() # on function call
... print globals().keys()
...
>>> class klass:
... def __init__(self):
... print locals() # on instance creation
... print globals().keys()
... print locals() # on class creation
... print globals().keys()
...
{'__init__': <function __init__ at 76ed00>}
['__builtins__', '__name__', 'func', '__doc__']
>>> func(1)
{'a': 2, 'x': 1}
['__builtins__', '__name__', 'func', 'klass', '__doc__']
>>> x = klass()
{'self': <klass instance at 76f8d0>, 'arg': None}
['__builtins__', '__name__', 'func', 'klass', '__doc__']
>>> def nester(L, M, N):
... class nested: # assigns class to name
... def __init__(self):
... pass
... print locals() # local=class global=mod
... print globals().keys() # no access to L/M/N!
... return nested
...
>>> nester(1, 2, 3)
{'__init__': <function __init__ at 761e30>}
['__doc__', 'nester', '__name__', 'x', 'func', 'klass',...]
<class nested at 762960>
♦ apply [now func(*args)] runs functions with argument tuples
♦ eval evaluates a Python expression code-string
♦ exec runs a Python statement code-string (3.X: exec())
♦ getattr fetches an object’s attribute by name string
♦ Supports run-time program construction
♦ Supports embedding Python in Python
Basic usage
>>> x = "2 ** 5"
>>> a = eval(x)
>>> a
32
>>> exec "print a / 2" # 3.X: exec('print(a / 2)')
16
>>> def echo(a, b, c): print a, b, c
...
>>> apply(echo, (1, 2, 3)) # now: echo(*(1, 2, 3))
1 2 3
>>> import string
>>> getattr(string, "uppercase")
'ABCDEFGHIJKLMNOPQRSTUVWXYZ'
>>> D = {}
>>> exec "import math\nx = math.pi" in D, D # 3.X exec()
>>> D['x']
3.14159265359
Example
♦ Import module by name, run function by name/args
♦ Runs: [ message.printer(‘sir’, ‘robin’) ]
♦ Also see: newer __import__ function
♦ Preview: will revisit model to embed Python in C
file: message.py
def printer(str1, str2):
return 'brave', str1, str2
file: dynamic.py
def runFunction(moduleName, functionName, argsTuple):
exec 'import ' + moduleName
module = eval(moduleName)
function = getattr(module, functionName)
return function(*argsTuple)
if __name__ == '__main__':
from sys import argv
print runFunction(argv[1], argv[2], tuple(argv[3:]))
Command line
% python dynamic.py message printer sir robin
('brave', 'sir', 'robin')
♦ time module contains C library type tools
♦ Useful for performance tweaking (along with profiler)
♦ Warning: be careful to compare apples to apples!
→ See also newer “timeit” module: automated, portable timing tools
>>> import timeit
>>> min(timeit.repeat(stmt="[x ** 2
for x in range(1000)]", number=1000, repeat=5))
0.5062382371756811
c:\code> python -m timeit -n 1000 -r 5 "[x ** 2 for x in range(1000)]"
1000 loops, best of 5: 505 usec per loop
Example: inline stack alternatives
♦ List based stacks
['spam0', 'spam1', 'spam2'] ←top
● Use list in-place changes: append/del
● Lists resized on demand: grown in increments
♦ Tuple-pair based stacks
top→ ('spam2', ('spam1', ('spam0',
None)))
● Use tuple packing/unpacking assignments
● Build a tree of 2-item tuples: (item, tree)
● Like Lisp ‘cons’ cells/linked lists: avoids copies
file: testinline.py
#!/opt/local/bin/python
import time
from sys import argv, exit
numtests = 20
try:
pushes, pops = eval(argv[1]), eval(argv[2])
except:
print 'usage: testinline.py <pushes> <pops>'; exit(1)
def test(reps, func):
start_cpu = time.clock()
for i in xrange(reps): # call N times
x = func()
return time.clock() - start_cpu
def inline1(): # builtin lists
x = []
for i in range(pushes): x.append('spam' + `i`)
for i in range(pops): del x[-1]
def inline2(): # builtin tuples
x = None
for i in range(pushes): x = ('spam' + `i`, x)
for i in range(pops): (top, x) = x
print 'lists: ', test(numtests, inline1) # run 20 times
print 'tuples:', test(numtests, inline2)
Results (on an ancient machine…)
% testinline.py 500 500 --20*(500 pushes + 500 pops)
lists: 0.77 --lists: append/del
tuples: 0.43 --tuples: pack/unpack
% testinline.py 1000 1000 --20K pushes + 20K pops
lists: 1.54
tuples: 0.87
% testinline.py 200 200
lists: 0.31
tuples: 0.17
% testinline.py 5000 5000
lists: 7.72
tuples: 4.5
Related Modules
datetime
>>> from datetime
import datetime, timedelta
>>> x = datetime(2004,
11, 21)
>>> y = datetime(2005,
3, 19)
>>>
>>> y - x
datetime.timedelta(118)
>>>
>>> x + timedelta(30)
datetime.datetime(2004, 12, 21, 0, 0)
profile
● Run as script or interactive, like pdb debugger
● As
script →
python -m profile script.py
● Warning: don’t run at IDLE prompt (too much data)!
● cProfile: more efficient version in 2.5+
● pstats to report on results later
● Optimizing: profile, time, shedskin/psyco, move to C
>>> import profile
>>> profile.run('import
test1')
35 function calls in 0.027 CPU seconds
Ordered by: standard name
ncalls tottime percall cumtime percall filename:lineno(function)
16
0.001 0.000 0.001
0.000 :0(range)
1
0.005 0.005 0.005
0.005 :0(setprofile)
1
0.002 0.002 0.022
0.022 <string>:1(?)
1
0.000 0.000 0.027
0.027 profile:0(import test1)
0
0.000 0.000 profile:0(profiler)
1
0.000 0.000 0.020
0.020 test1.py:2(?)
15
0.019 0.001 0.020
0.001 test1.py:2(myfun)
Example: timing iteration alternatives
Byte code
●
Modules compiled
to portable byte-code on import: .pyc
●
compileall module forces imports, to make .pyc’s
●
.pyo files created and run with –O command-line flag
(removed in 3.5!)
Frozen Binaries
●
Package byte-code
+ Python in an executable
●
Don’t require
Python to be installed
●
Protect your
program code
Other options
●
Import hooks
support zip files, decryption, etc.
●
Pickler converts objects to/from text stream (serializer)
|
Format |
Medium |
|
Source files |
.py files,
scripts |
|
Source files, no console |
.pyw files (Windows,GUI) |
|
Compiled byte-code files |
.pyc files, .pyo files(1.5 ~
3.4) |
|
C extensions on Windows |
.pyd (a .dll with init function) |
|
Encrypted byte-code files |
Import hooks, PyCrypto |
|
Frozen binaries, self-installers |
Py2Exe, PyInstaller,
cx_freeze |
|
distutils (see below), pip
(newer) |
Setup.py installation scripts |
|
Zip files of modules |
Auto in 2.4+ (zipimport
2.3+) |
|
Pickled objects |
raw objects |
|
Embedding mediums |
databases, etc. |
|
Jython: Java bytecode |
network downloads, etc. |
|
PyDoc |
Displaying docstrings,
program structure |
|
PyChecker |
Pre-run error checking (a “lint” for
Python) |
|
PyUnit |
Unit testing framework (a.k.a. unittest) |
|
Doctest |
docstring-based
regression test system |
|
IDEs |
IDLE, Komodo, PythonWin,
PythonWorks |
|
Profilers |
profile, hotshot |
|
Debuggers |
pdb,
IDLE point-and-click, print |
|
Optimization |
Psyco,
.pyo bytecode, C
extensions, SWIG |
|
Packaging |
Py2Exe, Installer, Freeze (above) |
|
Distutils |
packaging, install, build script system |
|
Language tools |
module packages, private attributes, class
exceptions, __name__==__main__, docstrings |
See also third-party testing tools, such
as Nose
Simplest convention
if __name__ == '__main__':
unit test code here...
doctest module
● automates interactive session
● regression test system
# file spams.py
"""
This module
works as follows:
>>>
spams(3)
'spamspamspam'
>>>
shrubbery()
'spamspamspamspam!!!'
"""
def spams(N):
return 'spam' * N
def shrubbery():
return spams(4) + '!!!'
if __name__ ==
'__main__':
import doctest
doctest.testmod()
C:\Python25>python
spams.py -v
Trying:
spams(3)
Expecting:
'spamspamspam'
ok
Trying:
shrubbery()
Expecting:
'spamspamspamspam!!!'
ok
2 items had no
tests:
__main__.shrubbery
__main__.spams
1 items passed
all tests:
2 tests in __main__
2 tests in 3
items.
2 passed and 0
failed.
Test passed.
unittest module (PyUnit)
● class structure for unit test code
● See library manual: test suites, …
import random
import unittest
class TestSequenceFunctions(unittest.TestCase):
def setUp(self):
self.seq =
range(10)
def testshuffle(self):
# make sure the shuffled sequence does
not lose any elements
random.shuffle(self.seq)
self.seq.sort()
self.assertEqual(self.seq, range(10))
def testchoice(self):
element = random.choice(self.seq)
self.assert_(element
in self.seq)
def testsample(self):
self.assertRaises(ValueError, random.sample, self.seq, 20)
for element in random.sample(self.seq, 5):
self.assert_(element
in self.seq)
if __name__ ==
'__main__':
unittest.main()
See also more recent “pip” installer
regime, and the PyPI site
# setup.py
from distutils.core import setup
setup(name='foo',
version='1.0',
py_modules=['foo'],
)
# Usage
python setup.py sdist # make source distribution
python setup.py install # install the system
python setup.py bdist_wininst # make Windows exe installer
python setup.py bdist_rpm # make Linux RPM installer
python setup.py register # register with PyPI site
# setup.py for C extension modules
from distutils.core import setup, Extension
setup(name='foo',
version='1.0',
ext_modules=[Extension('foo', ['foo.c'])],
)
♦ Built-ins
● Lists, dictionaries, strings, library modules, etc.
● High-level tools for simple, fast programming
♦ Python extensions
● Functions, classes, modules
● For adding extra features, and new object types
♦ C extensions
● C modules, C types
● For integrating external systems, optimizing components, customization
Click here to go to
lab exercises
Click here to go to
exercise solutions
Click here to go to
solution source files
Click here to go to
lecture example files