6. Modules






Why use modules?



     Code reuse

     System name-space partitioning

     Implementing shared services or data





Module topics


     The basics

     Import variations

     Reloading modules

     Design concepts

     Modules are objects

     Package imports

     Odds and ends

     Module gotchas




Module basics



     Creating modules:        Python files, C extensions; Java classes (Jython)

     Using modules:            import, from, reload(), 3.X: imp.reload()

     Module search path:     $PYTHONPATH





file: module1.py


def printer(x):           # module attribute

    print x




Module usage


% python

>>> import module1                    # get module

>>> module1.printer('Hello world!')

Hello world!


>>> from module1 import printer       # get an export

>>> printer('Hello world!')


>>> from module1 import *             # get all exports

>>> printer('Hello world!')







from *” can obscure variables’ meaning


>>> from module1 import *     # may overwrite my names

>>> from module2 import *     # no way to tell what we get

>>> from module3 import *

>>> . . .

>>> func()     # ?!


# Advice: use “from *” with at most 1 module per file







from” does not play well with “reload”


>>> from module1 import func       # copy variable out

>>> func()                         # test it


…change module1.py…


>>> from imp import reload         # required in 3.X

>>> reload(module1)                # FAILS: unbound name!


>>> import module1                 # must bind name here

>>> reload(module1)                # ok: loads new code

>>> func()                         # FAILS: old object!


>>> module1.func()                 # this works now


>>> from module1 import func       # so does this

>>> func()


# Advice: don’t do that--run scripts other ways





Module files are a namespace



     A single scope: local==global

     Module statements run on first import

     Top-level assignments create module attributes

     Module namespace: attribute ‘__dict__’, or dir()




file: module2.py

print 'starting to load...'


import sys

name = 42


def func(): pass


class klass: pass


print 'done loading.'




>>> import module2

starting to load...

done loading.


>>> module2.sys

<module 'sys'>

>>> module2.name


>>> module2.func, module2.klass

(<function func at 765f20>, <class klass at 76df60>)


>>> module2.__dict__.keys()   # add list() in 3.X

['__file__', 'name', '__name__', 'sys', '__doc__', '__builtins__', 'klass', 'func']





Name qualification



     Simple variables

      ‘X’ searches for name ‘X’ in current scopes


      ‘X.Y’ searches for attribute ‘Y’ in object ‘X’


      ‘X.Y.Z’ gives a path of objects to be searched


      Qualification works on all objects with attributes: modules, classes, built-in types, etc.










Import variants




     Module import model

      module loaded and run on first import or from

      running a module’s code creates its top-level names

      later import/from fetches already-loaded module



     import and from are assignments

      import assigns an entire module object to a name

      from assigns selected module attributes to names







import mod

fetch a module as a whole

from mod import name

fetch a specific name from a module

from mod import *

fetch all top-level names from a module


force a reload of module’s code
















Reloading modules



     Imports only load/run module code first time

     Later imports use already-loaded module

     reload function forces module code reload/rerun

     Allows programs to be changed without stopping

     3.X: must first “from imp import reload” to use!



General form


import module                # initial import

[use module.attributes]

...                          # change module file


from imp import reload       # required in 3.X

reload(module)               # get updated exports

[use module.attributes]




Usage details


     A function, not a statement

     Requires a module object, not a name

     Changes a module object in-place:

      runs module file’s new code in current namespace

      assignments replace top-level names with new values

      impacts all clients that use ‘import’ to fetch module

      impacts future ‘from’ clients (see earlier example)



Reload example (optional section)


     Changes and reload file without stopping Python

     Other common uses: GUI callbacks, embedded code, etc.



% cat changer.py

message = "First version"


def printer():

    print message



% python

>>> import changer

>>> changer.printer()

First version




[modify changer.py without stopping python]


% vi changer.py

% cat changer.py

message = "After editing"


def printer():

    print 'reloaded:', message



[back to the python interpreter/program]


>>> import changer

>>> changer.printer()      # no effect: uses loaded module

First version


>>> reload(changer)        # forces new code to load/run

<module 'changer'>

>>> changer.printer()

reloaded: After editing






Package imports


Module package imports name directory paths:

      Module name dir.dir.dir…” in import statements and reloads

      “import dir1.dir2.mod”       loads dir1\dir2\mod.py

      “from dir1.dir2.mod import name”

      dir1 must be contained by a directory on sys.path (‘.’, PYTHONPATH, etc.)

      Each dir must have “__init__.py”  file, possibly empty (till 3.3: optional)

      __init__.py gives directory’s namespace, can use __all__ for “from*”

      Simplifies path, disambiguates same-named modules files







import dir1.dir2.mod


      dir0 (container) must be listed on the module search path

      dir1 and dir2 both must contain an __init__.py file

      dir0 does not require an __init__.py











Why packages?




        __init__.py       (__init__ needed if dir in import)


        main.py           (“import util” finds here)







    sys3\                 (here or elsewhere)

        __init__.py       (your new code here)

        myfile.py         (“import util” depends on path)

                          (“import sys1.util” doesn’t)




Advanced: Relative import syntax (2.5+)


To enable in 2.X (standard in 3.X):


    from __future__ import absolute_import   # till 2.7, “from” stmt only


In code located in package folder pkg:


    import string    # skips pkg: finds the standard library's version


    from .string import name1, name2   # import names from pkg.string only

    from . import string               # import pkg.string




Advanced: Namespace packages (3.3+)



Extension to usual import algorithm: directories without __init__.py, located anywhere on path, checked for last, and used only if no normal module or package found at level: concatenation of all found becomes a virtual package for deeper imports.  Not yet used much in practice.


See Learning Python 5th Edition for more advanced package details.





Odds and ends


     Python 2.0+: “import module as name”

      Like “import module” + “name = module”

      Also good for packages: “import sys1.util as util


     Loading modules by name string

      exec(‘import ‘ + name)



     Modules are compiled to byte code on first import

      ‘.pyc’ files serve as recompile dependency

      compilation is automatic and hidden


     Data hiding is a convention

      Exports all names defined at the top-level of a module

      Special case: __all__ list gives names exported by “from *”

      Special case: “_X” names aren’t imported by a “from*”


     The __name__ == ‘__main__’ trick

      __name__ auto set to ‘__main__’ only when run as script

      allows modules to be imported and/or run

      Simplest unit test protocol, dual usage modes for code


file: runme.py


def tester():

    print "It's Christmas in Heaven"


if __name__ == '__main__':       # only when run

    tester()                     # not when imported


Usage modes


% python

>>> import runme

>>> runme.tester()

It's Christmas in Heaven


% python runme.py

It's Christmas in Heaven





Module design concepts



     Always in a module: interactive = module __main__

     Minimize module coupling: global variables

     Maximize module cohesion: unified purpose

     Modules should rarely change other module’s variables



X = 99                  # reader sees only this


import mod

print mod.X ** 2        # always okay to use


import mod

mod.X = 88              # almost always a Bad Idea!


import mod

result = mod.func(88)   # better: isolates coupling











Modules are objects: metaprograms



     A module which lists namespaces of other modules

     Special attributes: module.__name__, __file__, __dict__

     getattr(object, name) fetches attributes by string name

     Add to $PYTHONSTARTUP to preload automatically




File mydir.py

verbose = 1


def listing(module):

    if verbose:

        print "-"*30

        print ("name: %s file: %s" %

                 (module.__name__, module.__file__))

        print "-"*30


    count = 0

    for attr in module.__dict__.keys():      # scan names

        print "%02d) %s" % (count, attr),

        if attr[0:2] == "__":

            print "<built-in name>"          # skip specials


            print getattr(module, attr)      #__dict__[attr]

        count = count+1


    if verbose:

        print "-"*30

        print module.__name__, "has %d names" % count

        print "-"*30


if __name__ == "__main__":

    import mydir

    listing(mydir)      # self-test code: list myself




     Running the module on itself




C:\python> python mydir.py


name: mydir file: mydir.py


00) __file__ <built-in name>

01) __name__ <built-in name>

02) listing <function listing at 885450>

03) __doc__ <built-in name>

04) __builtins__ <built-in name>

05) verbose 1


mydir has 6 names






Another program about programs



     ‘exec’ runs strings of Python code

     os.system’ runs a system shell command

     __dict__ attribute is module namespace dictionary

     sys.modules’ is the loaded-module dictionary




file: fixer.py

editor = 'vi'  # your editor's name


def python(cmd):

    import __main__

    namespace = __main__.__dict__

    exec cmd in namespace, namespace  

    # 3.X: exec(cmd, namespace, namespace)


def edit(filename):

    import os

    os.system(editor + ' ' + filename)


def fix(modname):

    import sys                      # edit,(re)load

    edit(modname + '.py')

    if modname in sys.modules.keys():

        python('reload(' + modname + ')')


        python('import ' + modname)




% python

>>> from fixer import fix

>>> fix("spam")             # browse/edit, import by name


>>> spam.function()         # spam was imported in __main__

>>> fix("spam")             # edit and reload() by name


>>> spam.function()         # test new version of function





Module gotchas




“from” copies names but doesn’t link



X = 99

def printer(): print X




from nested1 import X, printer    # copy names out

X = 88                            # changes my "X" only!





import nested1             # get module as a whole

nested1.X = 88             # change nested1's X




% python nested2.py


% python nested3.py





Statement order matters at top-level


     Solution: put most immediate code at bottom of file

file: order1.py

func1()               # error: "func1" not yet assigned


def func1():

    print func2()     # okay:  "func2" looked up later


func1()               # error: "func2" not yet assihned


def func2():

    return "Hello"


func1()               # okay:  "func1" and "func2" assigned


Recursive “from” import gotchas


     Solution: use “import”, or “from” inside functions

file: recur1.py

X = 1

import recur2           # run recur2 now if doesn't exist

Y = 2


file recur2.py

from recur1 import X    # okay: "X" already assigned

from recur1 import Y    # error: "Y" not yet assigned



>>> import recur1

Traceback (innermost last):

  File "<stdin>", line 1, in ?

  File "recur1.py", line 2, in ?

    import recur2

  File "recur2.py", line 2, in ?

    from recur1 import Y    # error: "Y" not yet assigned

ImportError: cannot import name Y





“reload” may not impact “from” imports


     “reload” overwrites existing module object

     But “from” names have no link back to module

     Use “import” to make reloads more effective

import module        Þ module.X reflects module reloads

from module import X Þ X may not reflect module reloads!




“reload” isn’t applied transitively


     Use multiple “reloads” to update subcomponents

     Or use recursion to traverse import dependencies


See CD’s Extras\Code\Misc\reloadall.py





Optional reading: a shared stack module



     Manages a local stack, initialized on first import

     All importers share the same stack: single instance

     Stack accessed through exported functions

     Stack can hold any kind of object (heterogeneous)



file: stack1.py


stack = []                           # on first import

error = 'stack1.error'               # local exceptions


def push(obj):

    global stack                     # 'global' to change

    stack = [obj] + stack            # add item to front


def pop():    

    global stack

    if not stack:

        raise error, 'stack underflow'  # raise local error

    top, stack = stack[0], stack[1:]    # remove front item

    return top


def top():

    if not stack:                       # raise local error

        raise error, 'stack underflow'  # or let IndexError

    return stack[0]


def empty():      return not stack      # is the stack []?

def member(obj):  return obj in stack   # item in stack?

def item(offset): return stack[offset]  # index the stack

def length():     return len(stack)     # number entries

def dump():       print '<Stack:%s>' % stack





Using the stack module


% python

>>> import stack1

>>> for i in range(5): stack1.push(i)


>>> stack1.dump()

<Stack:[4, 3, 2, 1, 0]>



Sequence-like tools

>>> stack1.item(0), stack1.item(-1), stack1.length()

(4, 0, 5)

>>> stack1.pop(), stack1.top()

(4, 3)

>>> stack1.member(4), stack1.member(3)

(0, 1)

>>> for i in range(stack1.length()): print stack1.item(i),


3 2 1 0




>>> while not stack1.empty(): x = stack1.pop(),


>>> try:

...     stack1.pop()

... except stack1.error, message:

...     print message


stack underflow



Module clients

file: client1.py

from stack1 import *

push(123)                  # module-name not needed

result = pop()


file: client2.py

import stack1

if not stack1.empty():     # qualify by module name

    x = stack1.pop()

stack1.push(1.23)          # both clients share same stack




Lab Session 5


Click here to go to lab exercises

Click here to go to exercise solutions

Click here to go to solution source files

Click here to go to lecture example files